VOLUME 355 NUMBER 7 JULY ( WHOLE NUMBER RANSACTION OF THE AMERICAN MATHEMATICAL SOCIET EDITED BY Dan Abramovich Peter W. Bates Patricia E. Bauman William Beckner, Managing Editor Krzysztof Burdzy Tobias Colding Harold G. Diamond Sergey Fomin Lisa C. Jeffrey Alexander Nagel D. H. Phong Stewart Priddy Theodore Slaman Karen E. Smith Robert J. Stanton Daniel Tataru Abigail Thompson Robert F. Williams PROVIDENCE, RHODE ISLAND USA ISSN 0002-9947 Available electrol,i," Transactions of the American Mathematical Society This journal is devoted entirely to research in pure and applied mathematics. Submission information. Sec Information for Authors at the end of this issue. Publisher Item Identifier. The Publisher Item Identifier (PII) appears at the top of the first page of each article published in this journal. This alphanumeric string of characters uniquely identifies each article and can be used for future cataloging, searching, and electronic retrieval. Postings to the AMS website. Articles are posted to the AMS website individuallv after proof is returued frein authors and before appearing in an issue. Subscription information. Transactions of the American Mathematical Socety is published nlonthly. Beginning in January 1996 Transactions is accessible from -.ams. org/publ±cations/. Subscriptiou prices for Volume 355 (2003) are as follows: for paper delivery, $1490 list, $1192 institutional member, $1341 corporate member; for electronic delivery, $1341 list, $1073 iustitutional llleml»er, $1207 corporate nlenlber. Upon request, subscribers to paper delivery of tl,is journal are also entitled to receive electronic delivery. If ordering the paper version, add $39 for surface delivery outside the United States and India: $511 te lndia. Expedited delivery te destillatiols in North Alnerica is $48; elsewhere $144. For paper dclivery a late charge of 10% of the subscription price will be inlposed up(m orders receivcd frein llOlllll,llll{'rs after Jalmary 1 of thc subscription year. Back nUlllber information. For back issues sec . ams. org/bookstore. Subscriptious aud orders shouht be addressed te the Anlerican Mathenlatical Society, P.O. Box 845904. Boston. MA 02284-5904 USA. Ail orders must be accorapanied by pay- ment. Other correspondence should be addressed te 201 Charles Street, Providence, RI 1129114-2294 l'SA. Copying and reprinting. Material in this journal rnay be reproduced by any means for educational and scientific purposes without fee or permission with the exception of re- production by services that collect fees for delivery of documents and provided that the cus- tomary acknowledgment of the source is given. This consent does hot extend to other kinds of copying for general distribution, for advertising or promotional purposes, or for resale. [equests for permission for commercial use of material should be addressed to the Acqui- sitions Department. American Mathematical Society, 201 Charles Street. Providence, RI 02904-2294 USA. Requests can also be lnade by e-mail to reprint-perm±ssionams.org. Excluded from these provisions is material in articles for which the author holds copy- right. In such cases, requests for permission to use or reprint should be addressed directly to the author(s). (Copyright ownership is indicated in the notice in the lower right-hand corner of the first page of each article.) Transactions o] the Amemcan Mathematical Society is published monthly by the Amer- ican Mathematical Society at 201 Charles Street, Providence, RI 02904-2294 USA. Peri- odicals postage is paid at Providence. Rhode Island. Postmaster: Send address changes to Transactions, American Mathematical Society, 201 Charles Street, Providence, RI 02904- 2294 USA. @ 2003 by the American Mathematical Society. All rights reserved. This journal is indexed in Mathematical Reviews, Zentralblatt MA TH, Science Citation Index @, Science Citation IndexrM-Expanded, ISI Alerting Services TM, CompuMath Citation Index @, and Current Contents® / Physical, Chemical & Earth Sciences. Printed in the United States of America. ( The paper used in this journal is acid-free and falls within the guidelines established to ensure permanence and durability. 10 9 8 7 6 5 4 3 2 1 a a7 aa a a a TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY CONTENTS Vol. 355, No. 7 Whole No. 818 July 2003 Borislav Karaivanov, Pencho Petrushev, and Robert C. Sharpley, Algorithms for nonlinear piecewise polynomial approximation: Theoret- ical aspects .......................................................... 2585 JSrg Brendle, The ahnost-disjointness nmnber may have countable cofinality ............................................................ 2633 Alina Carlnen Cojocaru, Cyclicity of Chi elliptic curves modulo p ..... 2651 Tonghai Yang, Taylor expansion of an Eisenstein stries .................. 2663 Eric Freelnan, Systems of diagonal Diot)hantiim inequalities ............. 2675 Francisco Javier Callego and Bangere P. Purnaprajna, On the canonical rings of covers of surfaces of minimal degree ............... 2715 H. H. Brungs and N. I. Dubrovin, A classification and examples of tank one chain domains ................................................... 2733 Donald W. Barnes, Oll the si)ectral sequence constructors of Guichardet and Stefan ........................................................... 2755 Steven Lillywhite, Formality in an cquivariant setting ................... 27ïl Neil Hindman, Dona Strauss, and Yevhen Zelenyuk, Large rectangular semigroul)s in Stone-Cech compactifications .......................... 2795 Takehiko Yalnanouchi, Galois groups of quantum group actions and regularity of fixed-point algebras ..................................... 2813 Boo Riln Choe, Hyungwoon Koo, and Wayne Slnith, Composition operators acting on holomorphic Sobolev spaces ..................... 2829 B. Jakubczyk and M. Zhitolnirskii, Distributions of cora.nk 1 and their characteristic vector fields ........................................... 2857 E. Boeckx, When are the tangent sphcre bundles of a 1Rielnalmian ma.lfifold reducible? ........................................................... 2885 Henri Colnlnan, Criteria for large deviations ........................... 2905 Seung Jun Chang, Jae (Iii Choi, and David Skoug, Integration by parts fornmlas involving generalized Fourier-Feylnan transforms on flmction space ................................................................ 2925 Miehiko Yuri, Thermodynamic formalism for countable to one Markov systems .............................................................. 2949 D. G. De Figueiredo and Y. H. Ding, Strongly indefinite flmctionals and lnultiple solutions of elliptic systelns ............................. 2973 F. Rousset, Stability of sinall alnplitude boundary layers for lnixed hyperbolic-parabolic systenls ........................................ 2991 TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETh oluine 355, Number 7, Pages 2585 2631 S 0002-9947(03)03141-6 Article electronically published on March 19, 2003 ALGORITHMS FOR NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION: THEORETICAL ASPECTS BORISLAV KARAIVANOV. PENCHe PETRUSHEV, AND ROBERT C. SHARPLEY ABSTRACT. In this article algorithms are developed for nonlinear n-terre Courant element approximation of functions in Lp (0 < p <_ oo) on bounded polygonal domains in ll 2. Redundant collections of Courant elements, which are generated by multilevel nested triangulations allowing arbitrarilv sharp angles, are investigated. Scalable algorithms are derived for nonlinear approx- imation which both capture the rate of the best approximation and provide the basis for numerical inlplenlentation. Sinlple thrcsholding criteria enable approximation of a target ftlnction f te optinlal|y high asynlptotic rates which are deternlined and automatically achieved by the inherell| Sllloothlless of f. The algorithms provide direct approxinlation estimates and permit utilization of the general .lackson-Bernstein machinery te characterize n-terre Courant element approximation in terres of a scale of smoothness spaces (B-spaces) which govern the approximation rates. 1. INTRODUCTION Highly detailed Digital Terrain Elevation Data (DTED) and associated ilnagery are new becolning widely availal)le for lnOSt of the earth's surface. However, al- gorithms for effective approximatiol of data of this type are net yet available. A primary lnotivation for this work is the developlnent of effective algorithms for nonlinear piecewise polynolnial approximation of DTED maps frein a redundant hierarchial system over (possibly) irregular triangulations which are constructive in nature. Application of the ideas and theory flOln [4] te the resulting framework will permit optilnal entropy tree encoding of the elevation data, elmble progressive view-dependent refinements which lnay be focused te user-localized regions, and permit the registration of similarlv encoded ilnage textures te the surface (see [10], [4] for lnore details). Our philosophy is that dependable practical approximation procedures can be built only upon a solid theoretical basis. Accol'dingly, we have two primary goals in this paper. The first is te better understand nonlinear piecewise polynolnial Received by the editors May 2, 2002. 2000 Mathematics Subject Classification. Primary 41A17, 41A25, 65D18; Secondary 65D07. 42B35. Key words and phrases. Nested irregular triangulations, redundant representations, nonlinear n-terre approximation. Courant elements, Jackson and Bernstein estimates. The second and third authors were suppoïted in part by Grant NSF #DMS-0079549 and ONF( N00014-01-1-0515. AI1 three authors were supported in part by ONR grant N00014-00-l-04ï0. 2003 American Mathematical Societ. 2585 2586 B. KAIAIVANOV, P. PETRUSHEV, AND IR. C. SHAIPLEY approximation, in particular, to UlMerstand the natme of the global smoothness conditions (spa.ces) which govern the rate of approximation. The second goal is to develop or refine existing constructive approximation methods for nonlinear approx- ilnation which capture the rate of the best approximation and tan be implemented effectivelv in practice. This paper addresses nonlinear n-terre approximation by Courant elements gen- era.ted by multilevel nestcd triangulations, glore precisely, for a giwn bounded polygonal domain E C N 2. let (oE)0 be a sequence of triangnlations such that each level OE is a triangulation of E consisting of closcd triangles with disjoint interiors and a refillellll,ltt of tho in'crions level OEn-1- VP impose some lltild natu- rai conditions on the triangnlations in order to prevent possible deterioration, but out results arc valid for fairlv generM triangula.tions with sharp angles. define Z := U,20 OE,- Each such nmltilevel triangulation Z gencrates a ladder of spaces 0 C 81 C consisting of piecewise lincar flmctions, where ,, (m O) is spanned Iv ail Courant el'nelts ç0 SUl)ported on oeils 0 at the m-th level . Utilizing these primal elemems, we cmsider mmlinear approximation bv n-terre piecewise liner fmwtims of the form N = j= aç, where mav conte from diren levels mM lwatims. ()ur first gml is to «haracterize the approximation spa«es «onsisting f Ml fimctions wilh a giveu rate of aplroxinmtion. For approxi- matière in L v, p < , lhis is donc lu [1 Il, wh«re a collection of smoothness spaces (called B-spaees) was introdu«ed and ntilized, ht this 1)aper, we develop this theory in the more complicated case of approximation in the uniform norm (p = oe). Out progrmn consists f the following steps. First. in order to qumttify the approxima- tion process, we dvvelop a collection of smoothness spaces B (T) which depend on T and will govern the best approximation. Second. we prove compmfion Jackson and Bernstein estimates, mtd. third, we clmracterize the approximation spaces bv interpolation spa('e methods. (htr second and prinmry goM is, by using the B-spa«es mM the related tech- niques, to develop (or refine) Mgorithms for nonlinear n-terre Courront element al»proximation so that the new Mgorithms are capMfle of achieving the rate of the best approximation. In the present paper, we develop three such algorithms for -term Courant element approxintation in L v, which we call "'threshold'" (p < ), "'trim and cut'" (0 < p ), mM "'Imsh the error'" (p = ) algorithnts. The first step of ea«h of th«se algorithms is a d««omposition step. We denote by O the set of all cells (supports of Courant elements) generated bv (ç)eo is obviouslv redmMant mtd, therefore, every fmtction f lins infinitely many representations of the form (-) = (f)ç- It is crucial to have a suciently ecient (sparse) initial representation of the fimc- tion f çhat is being approximated. In our case. this means that the representation (1.1) of f should allow a reMization of the corresponding B-notre [[f[[B(OE). Thus the problem of obtaining an ecienç iniçial representation of the fimctions is tightly related ço çhe development of the -spaces. achieve such eciencv bv using good projecçors into the spaces , m = 0, 1 ..... For compleçeness and comparison, we first consider çhe naturM "threshold'" al- gorithm for -çerm Courront elemenç approxinmçion, which is valid only in 0 < p < .. This algorithm simply takes the largest (in Lp) n-terres from (1.1). NONLINEAR PIECEW'ISE POLYNOMIAL APP1ROXINIATION 2587 Usiug the results ri'oto [11], if is easy fo show that the "threshold'" atgorithm cap- turcs the rate of the best n-terre Courant element approximation in Lp (p < ). The second algorithm, which we call "trim aud cut", originates from the proof of the Jackson estimate in [7] and uses the following idca. First, we partition through a coloring into a fmnily of disjoint trees " (with respect to the inclusion relation): := U.= Second. we "trim'" each tree by removing cells 0 G " corresponding to iusiguificant small terres aoço from (1.1), located near the t, ips of the branches. Third, we divide ("cut") the remaining parts of each tree " into sections of small "energy". Finally, we rewrite the significant part of each section as a linear combination of a slnall number of Courant elements. The resulting terres determine the final approxilnmt. b shall show that "'triln and cut" is capable of achieving the rate of the best approximation iii Lp (0 < p ). Pivotal in our development is the "push the error" algorithm, the nmne of which was coined bv Nira Dvn. The idea for this algorithm appears in [5] and mav be roughly described in LOe as follows. For a fixed e > 0. we "'Imsh the error" with «, starting ffoto the coarsest level (-)0 and proceeding to finer levels. Nalnely, we denote by A0 the set of ail 0 (-)0 such that laol > e (11011 = 1) and define 0 := 0e aoço. Theu we rewrite ail remaining terres aoço at the next level and add the resultiug terlns to the existing terres aoço, 0 (-)1- We denote the uew terlns by doço, 0 Or, and select in A1 all 0 (-)1 SllCh that Idol > e. xx contime pushing the error in this wav to the finer levels in the lepreseutation of f. Finallv. we define our approxilnant bv A := k0 Aj. Thus terres doço with Idol ¢ are discarded only at a very fine level, and hence the error (in L) is e. Of course, this haire "push the error" algorithm calmot achieve the rate of the best. approximation. However, we shall show iii 3.3 and 5. after solne substan- tial improvelnents, the algorithm is capable of achieving the rate of convergence of the best n-terre Com'ant element aI)proximation in the uniform norm. A focal point of our developlnent is the characterization of the approximation spaces generated by the best n-terre Courant element approximation in L and the characterization of certain approximatiou spaces sociated with the three al- gorithms developed, which show that they capture the rate of convergence of the best approximation. The outline of the paper is as follows. In 2. We collect all facts needed regarding multilevel triangulations, local approximation, quasi-interpolmts, and B-spaces. In 3. we develop and explore the three algorithms for nonlinear n-terln Courant ele- ment approximation: "'t hreshold" algorit hm (in 3.1), "t riln and eut'" algorithm (in 3.2), and "push the error" algorithln (in 3.3). Section 4 is devoted to establishing 3ackson and Bernstein inequalities in order to study best n-terre Courant element approximation. In 5, we show that the three algorithlns capture the rate of the best n-terre Courant elelnent approximation and identifv the associated approxi- mation spaces B-spaces. In 6, we discuss some of the main issues of nonlinear Courant element approximation. postpone until the Appendix the proof of an impol'tant coloring lemma used in 3.2 for tree approximation in the "'trim and cut'" algorithm. For convenience, we use the convention that positive constants are denoted bv c, c,.., throughout and thev may vary at everv occurrence. The notation A B lneans that ClA B c2A. 2588 B. KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY 2. PRELIMINARIES In this section we collect ail the facts needed regarding nmltilevel triangulations. local approximation, quasi-interpolants, and other results whiçh were developed in [11] a.lld earlier papers. The essentials are presened for clarity but without proofs. 2.1. iangulations. By definition çan be represented as h« ration of a finite set of closed triangles with disjoint interiors: E = zeN, . shall alwavs assmne thal lhere exists an initial triangnlation of E of this fi)rm. call a mullilev«l lriaguh#io¢ «»f E with levels (OE,) if lhe following conditions are ful- tilh,d: (a) l':v¢'ry lev¢,l OE,, is a lmrtiti«m AeT,,, and OE,, c«msists ¢»f «h»sed lrialigles with disjoint interiors. (h) l'h¢' l¢,v«ls (E,,) ¢»f ç are n¢'sed, i.c.. OEn+l is a refinement of («) Ea«l li«mgle , ]las al I'ast lwo and at most 310 children (sub- I i'img[(,s) in OE,z+l, wh¢,re .1I 4 is a constant. (¢1) l'he valence X,. of each vertex e of anv triangle OE (the number ¢,f the trianglcs ff'oto OE,, that share t' as a verex) is at most .OE, where 3 is a COllStallt. (e) No-ha¢gig-v«rli««s «omtitio: Ne» vertex of anv triangle çm that I»elongs to lhe interior of E lies in the interior of an edge of another triangle from Xk' denote bv ç',,z the set of ail vertices of triangles from ,, where if v ç is on the bCmndary of E, we inclnde in ç as many copies of v as is its multiplicity. With this mderstanding, we set ç= now introduce three types of nnlltilevel nested triangulations which will play an essential role in our developments: Locally regular triangulations. \Ve call a multilevel triangulation 2- = [.J,n_>0 OE, of E, a compact 1)olygolml (lomain in N 2, a locally regular triangulation. or briefly an LR-triangulation. if T satisfics the following additional conditions: (i) Thcrc exist constants 0 < r < p < 1 (r ), such tllat for each L ç and anv child ' of L that belongs to ç. (2.2) rl I I'1 piPI. (ii) There exists a constant 0 3 i su«h that for each '. " ç (m 0) with a. common vertex. (2.3) la,, . Strong locally regular triangulations. call a nlultilevel triangulation T = ,0 T of E, a compact polygonal domain in OE2, a strong locally regular triangulation, or simply an SLR-triangulation. if ç satisfies condition (2.2) and also the following condition (which replaces (2.3)): NONLINEA1R PIECEWISE POLYNOMIAL APPROXIMATION 2589 (iii) AJfie transform angle condition: There exists a. constant /3 = /3(T) > 0 (0 < /3 < -) such that if 0 OE (m, > 0) and A R 2 R 2 is an ane 3 -- transtrm mapping A0 one-to-one, onto an equilateral reference triangle, then for every triangle A ¢ T with a. conlnlon vertex with Au, we have (2.4) rein angle (A(&)) 2 fl where A(&) is the image of & under A and is therefore also a triangle. Regular triangtalations. B.v &'finition a multil«-«el triangulati(m T of E C 2 is called a regular triangulation if ç sa.tisfies the fi»llowing condition: (iv) There exists a constanl ç = ff(T) > 0 su«h that the minimal angle of each T is greater than or equal to fl- The remainder of this subsection makes several observations to better mMerstand the natm'e of multilevel triangulations. First, it is clear that the classes of LR- and SLR-triangulations are each invm'iant ramer ane transforms. X> next observe that ea«h SLR.-tlimgulation is m LR-triangulation. but that the converse statement does hot hold. loreover. ea«h regular triangulation is an SI,R-triangulation, but again the converse is in general false. Çomtcrexalnples are given in [11]. Each type of triangulation dcpends on several parameters which are hot com- pletely independent. For instance, the paralneters of LR-triangulations are 3Io, , r, p, & and # (the caMinalitv of ). We could set al 1 = 7, P= 1-randelimi- nate these as parameters, but this would tend to obs«m'e the actual dependence of the estilnates upon given triangulations. We next brieflv dcscribe a siml)le standard procedm'e for conslructing lnultilevel triangulations. XX start ff'oto an initial triangulation of the given compact polygonal domain E C N . We then select a point on each edge of every triangle and join them within by edges to subdivide into four children. The collection of ail such children hecomes the first generation of triangles, which we denote bv . We recursively refine in this way to produce succeeding generations , ,.... The resulting collection T := U,0 OE, is a multilevel triangulation of E. It is important to know how the quantities ][, Inin angle(), and Inax edge() of a triangle T mv change as lnoves away froln a fixed triangle > within the saine level or through the nested refinements. Consider the case when ç is an LR-triangulation. Then conditions (i) and (ii) suggest a geometric rate of change of ]] (at the saine level). In fact, /ho rate is polynomial [11]. Fur/hermore, if '.A" OE (m 1) have a COnllnOl vertex and are also children of some T-I, then, as shoxw in [11], it is possible for ' to be equilateral (or close to such), but for " to have an uncontrollably sharp angle (see Figure 1). FIGU1RE 1. A skewed cell 2590 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY If2- is an SLR-tria.ngulatioi, the above configuration is impossible, but the trian- gles froln 2- still lnay bave uncontrollably shaip angles. In this case, rein angle() changes gradually froln one triangle to the adjacent ones. For anv vertex v G çm (m 0), we denote by 0 the cell at level m associated with v. i.e., 0.,. is the UlliOll of ail triangles from OE that have v as a common vertex. We denote by , the set of ail such cells 0 with v G ç, and set = Um0 oto. 2.2. Local piecewise linear approximation and qui-interpolants. d note by Hk the set of ail algebraic polynomials of total degree less than k. shall often refer to the following lelmna (see [11]), which estat)lishes the equivMence of differcnt norms of polynomials over different sers. Lemnla 2.1. Let P ll. k 1, and 0 < p,q . (a) For ang tr'iaogle C 2, (b) If A aod A' are tu,o trian91cs such tbat A' C 5 and I1 (c) tf a' c 5 .., la'l qll ,,,itn o < «, < 1, tben ) 1 1 Io tbe above e.rpressions, tbe coo.stants depeod at mo.st on tbe coespondm9 param- tiers and tbe constant c. l'he no-hanging-verlices condition (e) of triangulations guarantees the existence »f ('ourant elements. Namely, for any vertex v 6 ç (m 0) there exists a unique Courant element ç0, supported on 0 G , which is the unique continuous piecewise linear fllllCtion Oll E that is supported on 0, and satisfies ç0,. (v) = 1. We denote 6 := := (0)0¢0- We also denote by S the space of ail continuous piecewise linear flmctions over T. Clearly, S E S if and only ifs = S(v)ço.. Throughout the remainder of this section, we assume that T is an LR-triangulation of E. X shall often use the tbllowing stability estimates for Lennna 2.2. Let 0 < q attd S = ao ço, m O. witb coecients ao Then for every G Tre, we bave and hence IISIl, ( OEO,, ODA I 0 wzth constants of equivalence dependin9 only on the parameters of 2-. estimates the q-norm is replaced bg the sup-norm if q -- oe. In tbese The proof of this lemlna is fairly silnple and ca.n be found iii [11]. Local piecewise linear approximation. The local approximation by continu- ous piecewise linear functions will be an ilnportant tool in out flrther development. NONLINEAR PIECEWISE POLYNOMIAL APP1ROXIMATION 2591 For f E L(E), q > 0, and any A E "Ym (m _> 0), we denote the error of approximation to f from S by (2.5) (f) := (f, T), := inf I[/- SIIL(a), SSm where is the UlfiOn of all triangles rioto OE, that have a vert.ex in common with Quasi-interpolants. The set y of ail Courant elements is obviouslv redundant. To obtain a good (i.e., sparse) representation of a given fimction f, we shall use the following well-known quasi-interpolant: (2.6) Q,(f) := Q,,(f, 7) := (f, 0 where (f, 9) := lE f9 and (ç0) are the duals of (ç0) defined bv (2.7) « := AEOE.,ACO 9 the "central vertex'" with ,e the linear polynomial that is cqual to at 0, and equal to ,A at/he o/ber two vertices of (recall that N, is the valence of v). It is easily seen that (çe, e') = 500,, for 0.0' E . Obviously, Q is a linear projector, i.e., Q(S) = S for S E Sm. It is crucial that e E L and e is locally supported. Conquently, Q is locally bounded and provides good local approximation. Lemma 2.3. (a) If f Lv(E ). 1 5 q 5 , ad E ç, m OE O. then IIQ,(/)II.() cll/ll.(n)- (b) IfO < ç and g = e " P witb P E H2 and m O. tben IIQ.(g)llL.() clgll.(n), for The constats above depend oly on ] and the parameters of For a proof of this lenmm, see [11]. From the above lemlna, we see that Q,, : L,(E) -- S,, (1 _< l <- oe) is a locally bounded linear projector. There is a well-known scheme for extending Q to a nonlinear projector Q : Lo(E) S, for 0 < ri < 1. This is needed for nonlinear approximation in Lp (0 < p 1). To describe this extension, let PA.ç : Lo(A) H (0 < ) be a projector (linear if ri 1 and nonlinear if 0 < q < 1) such that IIf- P,(f)ll.() cE(f,A) for f E L(A), where E2(f, A) is the error of the best L.(A)-approxilnation to f froln H2 (the linear polynonlials). We define .(f) := and set (2.8) Z 11A. T,,o(f) := Qm(p,,o(f)), for f Lo(E ). 2592 Clearly, ,., 0< r/< 1). The lmXt lelnlna, established in [11], shows that Qm and Tm,., provide good local approximalions Dom 8. Lemma2.4. (a) ff f L,(E). 1 5 1 x, ad A ç, m O. the (b) lf f Lo(E ), 0 < 1 ,, artd A T, m 0. then lift- T,(fl)ll() c(fl)o. The coustats above doperai ozdy on 1 aztd the parametcrs of T. The needed convergence of Q,,,(f) and T(f) to f is provided bv tlle following res, dt (sec l,emma 2.15 ff'on, [11]). Lemma 2.5. {f f L,(E). th«t Now. we apply a well-known schtullo fi,r ohtaining sparse Courant elelnent re resentation of flm('tions. b defiue (2.9) qm := Q, - Q,n-t aud tre.-0 := T.o - T-L-o. for m R 0. where Q_ := 0 and T-l.o := 0. Clearly, qm(f), tm,o(f) m. For a given flmction f G L0(E ), 1 q OE. we define the sequence b(f) := (bo(f))oeo= flore the expression B. KAIRAIVANOV, P. PETIRUSHEV, AND IR. C. SHAIRPLEY L.,(E) -, 8m is a projector (linear if 1 >-- 1 and nonlinear if (2.10) q.m(f) =: bo(f)9o, m >_ O. Using Lennna 2.5. we have (2.11) f = q,(f)= bo(f)c2o in L,. m>_O m>_O OEO, If f C L,(E). 0 < '1 < 1. we define the sequence b,(f) := (bo.o(f))oeçm bv (2-1 ')) tm,o(f) =: bo.-o(f)o, m >_ O. 0E(9,. and again by Lemma 2.5, we have (2.13) m_>0 m_>0 Clearly. b(-) is a linear operator while b(-) (0 < r/< 1) is nonlinear. NONLINEAtl PIECEWISE POLYNOMIAL APPROXIMATION 2593 2.3. B-spaces. In this section, we include the necessary tools for the B-spaces which we need for nonlinear -t, erln ('ourant element aI)proxination. The B-spaces over multilevel nested triangulations of IR 2 are introduced in [11] and used there for nonlinear n-term Courant element approximation in L»(IR 2) (0 < p < oc). Iu the present paper, we shall use the B-spaces for n-terre Courant element approximation in Lp(E) (0 < p <_ o), where E is a compact polygonal domain in IR 2. We shall put the emphasis on approximation iu the lllliforlll llOrlll (p = Oo). There are three t.vpes of B-spaces (skim.v, slim, and fat B-spaces) that w('re introduced in [11] to serve different pnrposes. For ('om'ant (,lement apl»roximation, we need the slim B-spaces, which we shall simply call B-spaces. Throttghout this paper, we assume that "Y is an L/?-triangulation of a compact polygonal domain E in IR . Moreover, the B-spaces B(T), with parameter set 1/r := a + 1/p a.ccording to two specific choices: (a) p = oc and a _> 1: or (b) 0 < p < .;x and a > 0, will arise naturallv in out algorithms and error estimates. These spaces have several equivalcnt definitions, which we briefly descrihe. Definition of B(T) via local approximation. We (lefine B(T) as the set ) l/r where SA (f) is the error of L(f=x)-approxilnation (local) to f ff'oto $,,, for A (see (2.5)). It is readily seen tha.t If + g f5 + gSç with < := i,{r, and If + s]B; = IfIB for s e &. Hence l" ]Bi is a Selni-norm if r 1 and a semi-quasi-norm if r < 1. By ThcoreIns 2.7 and 2.9 bclow, it follows that if f B(ç), then f Lp(E). Therefore. il is naturat to dcfine a (quasi-)norm in B (ç) by More generally, for (I < /< P, we &,fine (2.1Q Na,(/, ) := II/ll + (lll/v-/a(f)) Evidently, N«(f, T) = IIflIB(z). Vhel, clear from the context, we use . Definition of norm in B(E) via atomic deconlposition. For f L(E), we define ( (2.17) N(f) := inf (IOl-llcoollV Y=0eo coco where the iIfiInUm is takeu over all represent.alions f = 0eo c00 in L(E). Note t.hat the existeuce of such representations of f follows by (2.11) and (2.13). By Theorem 2.7 below, (IOl-llcooll) < implies Icoo(')l < , 0 and hence f c= Lp(E) and the series oo Ic00(.)l converges a.e. and in Lp(E). Therefore, the way in which the terres of the series are ordered is hot essential. 2594 B. KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY (2.18) If p = oc, then and the convergence in L-(E) implies a stronger (absolute) convergence in Lp(E) (- < p). Bx Lemma 2.1, it follows that N(f) inf ((o]l/p]co])) 1/r- f:0eo coco 00 k / f=eo c0 N,» (f) inf Icol"- f=-oeo cotpo Definition of norms in B(T) via projectors. For f Lo(E ). we let (2.19) f = bo.,(f)ço I,e lhc repreSelllatiol ,,f f ri'oto (2.11) if q 1 and from (2.13) if 0 < ç < 1. defiue (2.2o) x.(f) := (o-llbo«(f)çol) 0 and. more generally (in aceordanee with (2.16)). (2.21) NO,o(A) := (lOl/P-/°llbo.o(f)çollo) By Lemmas 2.1 aud 2.2, we have (2.22) No.o(f) (IAl/-/°llq(f)llL.)y if ç 1, (2.23) NQ.o(f) (IAIVP/°llt.o(f)llL,,¢a)) r if 0 < q < 1. and (2.24) NQ,,(f)(oe(lOI1/p[bo,,(f)l)r ) In the most interesting case of p = (2.25) General B-spaces. A more general B-space Bq(T). c > O. 0 < p. q < oc.. is defined as the set of all f E Lp(E) su«h that f=-oeo coco OeO. 2- 1o1<2 -m+l NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2595 where the fq-nOrln is replaced by the sup-nonn if q = oe. In this paper, we do hot need the B-spaces in such generality. Embedding theorems and equivalence of norms. e recall our assump- tions. have 0 < p 5 oe, and 1 if p = oe and a > 0 if p < oe. In both cases, 1/w := + 1/p (1/w := if p = oe). We record estilnat.es and embeddings froln [11], along with the necessary lno(lifications, which me necessarv for the development of the main results of this I)at)('r. Tho first oml)ed(ling rosult al)t)ears as Theorem 2.16 in [111. Theorem 2.6. For O < r < p or p = oe, r 1. tben for ar sequence of real unbers (co)oe. we bave (2.26) I«lç « I1«11; , OGO P - whe c depends onlg o . p. and tbe parmneters of Theorem 2.7. lf f Lu(E ) with O < l < p. and NQ.(f) < oe, then f Lv(E) (f C(E) if p = oe), ad f bas tbe repsetation f series com,er9in.q absolut,'ly a.e. i E and in Lp (respeetivelg. in C(E)). (2.27) IIfllp ll lbo.n(f), oll wheve c is idependent of f . Pvoof. For 0 < p < oe, the result follows from (2.11). (2.13), and Theoreln 2.9 below. If p = oe, the theorem follows by (2.11), (2.13). (2.25), and the following estimates: bo,v(f),ço Ibo,v(f)l Remark 2.8. It is easilv seen that Theorem 2.7 is hot true when p = and a < 1. For this re&son we impose the restriction a 1 when p = throughout. Theorem 2.9. The noms ][. ]]B7(), Na,v(-) (0 < V < P), N(.), a,,d NQ.v(.) (0 < ç < p), defined in (2.15). (2.16), (2.17). and (2.21). are equivale, t with constants of equivalence depending only on p. Proof. One proceeds exactly as in [11] (see the proof of Theorem 2.17 of that reference) and proves that l/l> ( (lll/v-/'aa(f),)) / f=o¢e coo O o provided 0 < W < p. To obtain the norm estimate from these semi-norn equiv- alences, we use Theorem 2.6 to give I111 c(). çsing this, (2.28), a,d the renmrk after the definition of N(f) iii (2.17), we obtain 2596 B KAPAIVANOV, P PETIRUSHEV, AND 1R. C. SHA1RPLEY For the reverse inequa.lity, we use Lemlna 2.2. Theorem 2.6, and (2.28) to obtain /- ) /P (o "bo,(f)o";) c(o, ,p) (o "bo.(f)o" c < cl[fHp+«( cfe(z. This and (2.28) iml,ly ,,(f) c[Iflla(z). l'he next embedding tlworem of Sobolev type foliotes immediately from (2.18) «»1" (2.24). Theoreln 2.10. kbr 0 < no < « and ri := «ontinmms cmb«ddin 9 "" T) (2.'2) B7)(7) c B ( , Interpolation. \Ve first recall some basic definitions from the real interpolation method. ], refcr the rea,l«r to [2] and [1] as gencral references for interpolation thcorv. For a pair of quasi-normed spaces Xo, Xt. emhedded in a Hausdorff space, the space X0 + X is defined as the collection of all flmctions f that can be repr sented as f0 + f with f0 X0 and f X. The quasi-norln in X0 + X is defined IIfl[x+x, := [If[xo+xl + i,,f ]lf011x + I[fxll.x,. f:fo+fl The K-functional is defined for each f Xo + X and t > 0 by (2.30) /x-(f,t) := Iç(f,t;Xo.X):= inf IlY011x0 +tllflll.xl- f:fo-}- fl The ïeal interpolation space (Xo, X), with 0 < A < 1 and 0 < q x is defined as the set of ail f Xo + X sueh that (1 )l where the Lq-norm is replaced bv the sup-norm if q It is easily seen that if X C X0 (X contimously embedded in Xo), then K(y.t) IlYllx for y x0 a.d t 1, .d. consequently, (2.31) ]f(Xo,X,), ]IYlIx + [2"Xlç(f.2-")] q Theorem 2.11. S'appose 0 < p and fuether assume that bolh o, 1 in the case p = .. and o, > 0 otherwise. Furthermore. let := (oj + l/p) -. j = O. 1. (2.32) NONLINEAR PIECEVvlSE POLYNOM1AL APPROXIMATION 2597 with «quival«rd hOtms, provid«d a = (1- A)ao + Aa with 0 < A < 1 and - := (et + l/p)-'. Proof. We shall prove (2.32) only in the case p > 1. For a proof of (2.32) when p _< 1, see [3]. We shall use the abbreviated notation B := B(7-) and B% := B- (7-), j = (}, 1. Also, we denote by gq the space of all sequences a = (a0)0e of real numbers such that Ila'l«. := (E I1 ) '/" < - We set / := 1 and non,,alizc the C(mrm,t elemenls iii Lp, tl,at is, II0]lp -- 1. we also renorlnalize the duals qS0 rioto (2.7) accordingly. \Ve denote again by b(f) = (bo)oee) the sequence ri'oto (2.10) witl, respect to the ,on,,alized Courant elelnents. By (2.24), Theorem 2.7, and Theorem 2.9, if f B , j = 0.1. then (2.33) f = Z bo(f)2o and recalling that thc clemcnts ç)0 arc normalizcd ill Lp. Thc corresponding statemcnt holds for functiols f G B a as wcll. b shall next employ the following intcli)olatiol theorcm (sec. e.g., 5.1 of [1] or [2]) which follows dircctly froln the dcfinition of thc N-fimctional and the llOrlllS of the interpolation spaces. Suppose T is a lincar operator which bomdedly maps X0 into tb and X into }, where (X0,-¥) and (lb, }) are couples of quasi-normed spaces as above. Then for 0 < A < 1 and 0 < q , T boundcdly lllaps (-¥o, X1),q illt.o (}b, }),q- Xb introduce lincar Opcl'ators Z and P a.s follows: Z is dcfincd by Z(f)o := bo(f), 0 G . and P is givcn l)y P(a) := oeoaoço, a = (ao)oeo. By (2.33), IIb(f)ll«., cllfllB for f e , j = 0.1. and hcnce Z B" g (boundcdly). Bv the above-lnenIiolmd interpolat ion t heol'eln. (2.34) Z. (B °,B)« (go,()« (boundedly). Similarly, if a G (,-, then bv Theorelns 2.7 and 2.9. we mav conclude that 7)(a) L,,= Y0eO a0ç0 is well defined. So if we set f = 7)(a), then [[P(a)['B <_ c inf [[(co)o,,e. <_ c,,a[[t.. 3 = 0.1. f=}2oee cooo Thus 7 ) " g'5 --' B (boundedly). and by interpolati(m (2.35) 7)" (gro" ç,)),r - ( B'°. B*))« (boundedly). Finally, we recall the well-known interpolation result (see, e.g.. [2], [1])" (2.36) (Otro,(rl),Mr = tr, where 1 _-- 1-k _1_ with 0 < A < 1. T T 0 T 1 Clearly, (2.32) follows by (2.33)-(2.36). Skinny B-spaces. The skimy B-spaces were introduced in [11] and used for characterization of nolflinear (discontinuous) piecewise polynolnial approxilnation on IR 2. \Ve next adapt that defilfition to the case of approximation on a compact polygonal dolnain E C ]R 2. Supposc 7- is a multilevel nested triangulation of E whicl, additionally satisfies condition, (2.2) (sec §2.1 and [11]). The skimlv B-space 2598 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY Bk(T), vhere k _> 1 and c and "r are as above, is defined as the set of ail f Lr(E) such that ) l/r (.aT) s(, :: (ç(-(s.))" < , where w(f, ) is a kth modulus of smoothness of f in L(), defined by wk(f,)r := sui, II(f.)ll.() and (f,-) is the kth difference of f. The norm in (T) is defined by * Fat B-spaces: The iink to Besov spaces. Sul)l)OSe T is an SLR-triangulation of a compact l)olygonal donmin E C 2. Similarly as in [11], we define the fat B- spacc (T), whcre k 1 an(1 a an(l r are as al)ove, as the set of all fimctions J L(E) such tha.t (-asl II(z := (ll-(I.a))" < oe- thcorcm, it rca(lilv fi,ll(,ws that Cl2(f, L) &(f) c2w2(f, &)r, and hence ]f8/=(7) clfIBv(T) clf]=(7). The st)ace 2(T) is a natural candidate to rcplace B (T) in nonlilmar -tcrm Courant element approximation. This is, how- evcr. only possible for suciently snlal] a (0 < a < a0). Otherwise 2(T) is too "'fat" and cannot do the job. Finally, we note that if T is a regular triangulation and 0 < a < k, thcn a(T) coincides with thc Besov space B(L). For a more complete discussion of this and other related issues, see [11]. 3. ALGORITHMS FOR n-TERM COURANT APPROXIMATION Decomposition step for ail approximation algorithlns. The first step of each of the three at)proxilnation algorithms that we consider in this section is a decolnl)osition step. This stcp is hot trivial, since the set (I'7- := (0)0eo of ail Courant elements is redundant and. therefore, each function has infinitely many representations using Courant elements. For each algorithm, it is crucial to have a sufficientlv efficient initial representation of the fimction f that is being approximated. This means that the representation of f should allow a realization of the corresponding B-norm. To construct the initial representation, we consider two cases of metric approxi- mation. If the approximation takes place in L u, 1 < p <_ :x, we utilize the decoln- position of f via quasi-interpolation from (2.11) with 1 <_ q < p, while if 0 < p < 1, we use (2.13) with 0 < 1 < p. In both cases, we bave an initial desirable sparse representation of f of the form (3.1) f = Eb°ç°' bo = bo(f), which allows a realization of the B-norm (see (2.24) (2.25), and Theorem 2.9). For the remainder of this sectiou, iii order to more easily track the dependency of the NONLINEAIR PIECEWlSE POLYNOMIAL APPROXIMATION 2599 constants appearing in the inequalities, we redefine Il fllB(ï) by (3.2) II/IIBa() which is an equivalent norm in B (T) (see Theoreln 2.9). Without loss of generality, we may aSsulne (when needed) that there is a final level (-)L (L < oe) in (3.1). 3.1. "Threshold" algorithln (p < oe only). In this algorithln we utilize the usual thresholding strategy used for o-erm allroxilllation ri'oin a basis in Lp (1 p < oe). The resulting procedure performs extremely well. due to the sparse re resentation realized bv the first step. k" note, llowever, that the derived error estimates involve COllstants that depend on p and becolne Ullbounded as p The "push the error'" and "trim and cut" algorithlns desçribed later in this section will be shown to achieve the correspollding estilnates for the Ulfifornl IIH'III (p = ). For this subsection we therefore assulne that f Description of the "threshold" algorithm. Step 1. (Decompose) Wc ue the dec(,nll)osition of f E Lp(E) froln (3.1). Step 2. (Select the largest terres) b order the terres (boo)o6o in a sequence (bo2 ço) so that (3.3) Ilbolço, ll IIboçoll ... Then we define the approximant T T n .4. (y)v y a= eçe. An (f), := Error estimation for the "threshold'" algorithln. denote the corre- sponding error of approxinmtiol of this threshold a.lgorithnx bv T T The argument used in establishing the Jackson error estimate in [11] may be mod- ified in obvious wvs to prove the following error estilnate. Theorem 3.1. IffG B(T). >0. 1/ç::+l/p (O<p<oe), then (3.a) T where c depends on . p. and the parameters of T. In 5, we shall need the following result: Lemma 3.2. If f : o boço is the decomposition off from (3.1). then j- 1 where (boço)= is as in Step 2 and c depends on . p. ad the parameters of T. Proof. Applying Theorem 3.4 from [11] to (bo9o)=,+ immediatelv provides the desired result. Remark 3.3. As we have mentioned, the main drawback of the "threshold'" algo- rithln is that it is hot applicable to approxilnation in the unifornl norm, since the constant c = c(c, p) in (3.4) tends to infilfity as p - oe and the performance of the algorithln deteriorates as p gets large. The obvious reason for this behavior is that 2600 B. IxAIRAIVANOV. P. PETRUSHEV, AND IR. C. SHARPLEY f can be built out of many terres (boço) which have small coefficieuts and are sup- ported at the smne location. These terres can pile up to an essential coutribution. but the algorithm will rail to antieipate their fiture significance. 3.2. "im and eut (the tree)" algorithm. The idea of this algorithIn h its origins in the proof of the Jackson estimate in [7] (see 5. pages 272-276). The approxima.tion considered there is hv wavelets or splines over a uniform partition in the unifornl llOrlil. shall refillC this i(lca to devclop ail algorithm for n-terre Colalt cleinent apl»I'OXilnatiol in Lv(E), 1 < p , over L-triangulations. begin with a ln'ief desci'iptiol of lhe algorithm and then elaborate on the details of «a«h of the inain steps. Description of the "triln and eut" algorithm. Step 1. (De«ompose). ki, us« thé, «olmnon decompositiou of f Lp(E) given ill (3.1). Step 2. ( ()ylonize th« colis q[" O into man9eable trees "). develop an al- gorilhm (pro«e«Iure) for ««»l»ring Ihe cells »f (-) in such a wav that the colis of the smne colon" fi»rm a Ir«e structure as deseribed in Lelmna 3.4 I»«l«»w. This organizaIim greatly simplifies Ihe lnanagement of the es- limaIes, b«»lh Ihe aI»proximalion coustru«tion and the enumeration of "active" Courant elelnents in our ai)proximant. Step 3. (Trim «ach tr«e). Sin«e all the elements lnay iuitiallv affect the B- spaee n«»rm of a flmction, we need to preprocess each tree by pruning alI t»ranehes whieh mav havc manv leas, but do hot make a significant contribution to thc norm of the flmctiou f, X do this bv running a stopping tilne argulnent from the finest level to a coarser level, until a significant cmnulative comribution is met. prune the branch just below that element. Step 4. (Partition the remainin9 trees into "'segments"). We continue to par- tition the remainders of each of the Iç trees bv cutting them at each of the joins of branches to form chains rioin the tree. We will easilv be al»le to track the numi)er of chains produced bv this procedure. A second stoI»ping time argument is then applied to eut the chains iuto "segments'" in ord«r to coutrol the uumber of significant eleluents added to the approximant (at most N0 + 1 from each segmeut) and to guaran- tee that the culnulative effeçt of the left-over elements (i.e., error) tan be controlled bv the final Step 5. Step 5. (Rewrite the "'se9ments'" fo control error). Here each segment is rewrit- ten at its fiuest level, and its terlninal element (with the new coefficients) and some of its neighboring elements are added fo the approximant. This allows for a void to be created, so that the residual of the segment will have disjoint support with all remaining segments well as the residuals of those previously processed. This insures that. the cumula- tive pointwise error remains under control. We now describe these rather vague steps in more detail. our earlier discussion. Step 1 is clear from Step 2. In the followiug lemnla, we construct a procedure for coloring the elelnents of (3 with K colors u, so that no two Courant elements of the saine color from the NONLINEAR PIECEW'ISE POLYNOMIAL APPROXIMATION 2601 saine level have SUl)l)orts that intersect; in fact, cor1espondillg cclls of the sanie color will bave a tree structure with set inclusion as the order relation. This allows us to partition into a disjoint milan of sets " (1 u Iç), and correspondingly K organize f as the sure f = =lf, where f := 0eo" boço. b tan then proceed to process each of the f, without vorrying about its terres from the saine level overlapping, and at worst a factor of K will came into the constants for the estimates that we dcrive. For its proof, sec tllc Appendix. Lemlna 3.4 (Coloring lcnuna). For ay multilevel-triaogulation T of E, lhe set := (T) of all cells geterated bg T con be represc,lcd as a .fitite disjoint io, of its svbsets (O"))1 with Iç = Iç(N0, M0) (No is te maximal valence ad 310 is the maximal umber of childrn of a triangle la T) such that each 0'" bas a lree structure with respect to set inclusion, i.e., if 0 . O" wzth (0) ° (0") ° O. the either 0 C O" or O" C 0 . In order to COml)lcte the remaining Stcps 3-5 we must consider two varialions iii the details of the algorithln, (lepell(ling on whether p = or 0 < 1 » < Thc case of thc mfiform mctric is plcscnt(,d in Subsecti(m 3.2.1, while thc case of Lp (0 < p < OE,) is giron in Subscction 3.2.2. 3.2.1. The ca.se p = ,. Fix e > 0 and let e* .-'- , « whcre we recall that Iç is the nmllber of colors rcpl'esenting the tlee structures. Step 3. Trimming of 0'" (1 oe Iç) with e*. triln each ", starting ff'oin the finest level and procecding to the coarsest level. remove ff'oto " everv cell 0 ° snch that (3.5) Ibol *. 0C0 ° denote bv F" the set of ail 0 " that bave been retained after completing this procedure, and by Fï the set of all final cells in F , i.e., 0 Fï iff there is no 0 F such that 0 Ç 0 . Clearly, for each 0 Fï, (a.6) Ibol for each 0' 0 . but Ibol > *. 0C0 0C0 denote fr := 0er boço. Therefore (3.) II/ -/11 max II booll mx Ibol *, 0°F 0C0 ' 0°F" 0C0"- K and hence, if we set ff := = fr, then (a.8) II/-/11 tç* = /oe- Step 4. Partitioning lhe branches of each tree F v into chains aod lhe chains into "'segments". For each of the tree structures F" (1 <_ , _< Iç). we denote by F the set of all branching cells in F" (cells with 1note than o11e child in F') and by Fo" h the set of all chain cells in F" (cells with exactly ana child in F'). It is easv to see that (3.) #r < In fact, o11e proceeds bv induction ff'oin the filleSt o coarser levels, associatillg each branch cell ff-oin F by a cell froln F}. For each branch cell, there is alwavs at least 2602 B. KARAIVANOV, P. PETF{USHEV, AND R. C. SHAF{PLEY one meniber of F} still available flore each descendant edge. Only one is used to associa.te with the current i)ranch cell, thereby leaying at least one available for its Imxt ancestor bran«h cell in that line. On the other hand, #Fh may be nmch lm'ger than #Fï, and so we will need to process these elenmnts. A collection of cells 01 D 02 D --- D 01 is called a chain if ibr j : 1 ..... ! - 1. Oj+ is a child of Oj and 0 Fn, and the terminal cell 0t Fï P. partition the tree F into chains. Namely, we start at the coarsest level and construçt (maximal) chains whiçh will ternfinate with either a final cell (in l'ï) or a branching cell (in F). XX contime this procedure to the finest level. Xb nex[ "section" each «hain into segments using e* as a threshold. Namely, if A is a chain and A : (03 )=1 with 01 D 02 D "-" D Or, then we start frOlll the coarsest eh'ment 01 mM smn the «oecients of each cell, moving to the next child of the «hain mtil the sure exceeds the threshold. At this point we cut the chain to form the first (significalt) segment and start this procedure again with the next child in line mitil this is hot possible (i.e.. ending without the threshold being crossed). X «ail this type of segment a "r«mn«nt segment". Therefine, this procedure cuts A into disioint segnwnts a »f thc finm (OjV +u ,j=i, I t O. so that each segment satisfies exactlv one of the following conditions: (a) a «onsists of a siugl« «significant et,Il"" (3.IO) (6) 13.11) (3.12) [bo, ] > e* (case ,,f p = 0), IV, I > * (c.e of # > 0), j=i I , Ibo, ' < i+p \Ve denote bv Z ' the set of all such seglnents o- = (Oj)j= i procedure. resulting [rOlll this Step 5. Reu,riting elcments from certain segments of '. Let a = (Oj)j= 1 be alw seginent ffOln Z, and suppose that the finest cell Op of rewrite the ('OUlalt elelnents ()=l boço) of the segment at its finest (m-th) hwel. finding coecient.s (co) such that COçO : bojço on 0. OOm,O°O,#O 1=1 XXdenote.« := {0 O : 0 0 # 0and0 C 0}. Obviously. ifa = 1 (i.e., the segment consists of a single cell), then the coecient relnains ulmhanged and = a = {01 }. Observe in any case that # N0 + 1 and Uoex o c o. Finallv. set := =1 , and correspondingly define (3.13) AZc(f) := aEN OEA as our approximant produced by the "trim and eut" algorithln. NONLINEAR PIECEWISE POLYNOMIAL APPFIOXIMATION 2603 Error estinlation for the "trilll and cut" algorithm (case p = oc). Suppose that the "triln and cut" procedure has been applied to a fmlction f with 4TC( ) > 0, and . e -f = 0A c090 is the resulting al)proximalt from (3.13), wheIe A« = U«ez ";» We denote ,,(e) := ,,±(«):= #A, and TC A.()(f) := IIf - AyC(f)[[, ln TC(f)oc := illf { TU } A()(f) "n(e) _< n . Note that each of these quantities depend ilnplicitly Oll 7. To COlnl)lete Ollr results for the "'tliln and cllt'" algorithln, we show fiI'St in Lelllllla. 3.5 that this is a good approxiInation to f, and then that the lmnfl)er of elelnents that are used in the approxillmnt satisfies the correct estinmtes (see Theol'eln 3.7 below). Lennna 3.5. Suppoe tlat ATC(f) is tle appro.rirnant for f given in equalion (3.13) whi«h bas been constr«'ted using the "'trm ood c,l'" algorithm. Then (3.14) Proof. Followil,g the definitiol, (3.13) of AyC(f), we (l('fille A":= coço. c,'E" 0 A', Then obviouslv. AC(f) = Eu=l/ A". SillCP g* - 2-,e it suces to shoxv that In Step 5 we extracted the heart of each segment a = (Oj)j=, added its con- tribution to tlle approxilnant (3.13), and cleared rooln for descendant cells. To estilnate the sociated error, we introduce tlle ring for a := 0 0,; then R« = 0 when a consists of a significant cell (i.e., condition (3.10) hokls). For any llOlmmpty ring « ( "), set ' := (0) and observe that at worst L(01) It is easv to see that all rings R« (a ") are disjoint and the set where A" mav differ froln fr is contailmd in «e£ and then over all colors u, it follows that K lift rc -4 (f)l, llf- This together with estimale (3.8) ilnplies the desired error estimale (a.l). Remark 3.6. Conditions (3.5), (3.11), and (3.12) can be relaxed by replacing every smn Ibo[ bv I[ boço Iloe. This would hot change the rate of approxilnation but lllay ilnprove the constants in a practical ilnplenmntation. 2604 B. KAFIAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY Theorem 3.7. lffEB(T),a_> 1. ç:=l/a, lhenforeachs>O, (3.16) TC A.(«)(f)oe _ ¢ and n(¢) whcre c = c(Sç, ,lv, «). Therefore, Pro@ have ah'cadv shown in Lelmna 3.5 that NTC ¢ < ¢: so we onlv need to estal,lish («) 5 c e-'llflla(). first obs,rve that it is cnough to estimate VEr, since «ontributions to thc approximmt oc«ur onlv as ea«h segment from is processed. Note that at most olp clement is «ontributcd for segments consisting »fa singlc signifi«ant eell (3.10) and at most o + 1 «ontributions for the segmets satisfying instead cither (3.11) «)r (3.12). I ordcr to estilnatc #Eu we first estilnate #F Ï, sin«e it will estilnate certain Icrms. Thc stq)ling criterimn (3.6) in Stcp 3. t3.1s) * < Ibol. 0C0 o must hold t«»r each 0'» I" Ï. No if wc al»lfly the ç-th power to both sides, use the cml)(,lding of thc scqu«,nc, spa«es (ç 1), smn over all 0 Fï, and observe that Ihe snl)l»orls of th« ('«lls in I') havc di}oint interiors, thon we obtain (3.)) #I'ï (e*) < l'hc rightmost incquality follows immediatclv bv our dcfinition of the nornl of B;'(ç) (ce (a.2)). To conlplete the proof of the theorem, we OlllV nced to cstablish a similar estimate tÇr Ihe lmmbcr of elcmcnts of E u. Rc«all, hoV(,l', that the segments « are %rmed as disjoint segmcnts of cells ff'oto thc tree structure and eome as one of two types. Esig. thosc cx('ceding the thrcshold (sec conditions (3.10) or (3.11)) and. Erem. those that do hot (sec condition (3.12)). KlOm thc construction it follows that renmant scgmcnts terminate with (qthcr a unique final «cll or a torique bran«hing oeil. and so bv (3.9), (a.0) #Zrcm #r + #Fï 'e #Fï, which has just l)een shown in (3.19) to satisfv the dcsired bomd. Thel'C%rc wc arc reduced to estimating timating #F Ï (sec (3.18)-(3.19)) mav be elnploycd once again. Indeed. we just replace the condition (3.18} with (3.21) e* < lbol, and use the fact that the segments are disjoint (considcred as part of the tree structure), in order to obtain (3.22) g (*) < lb01 r Although hot required hele, the following lemma will be needed in 5 and tan nov be established using the techniques of this section. NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2605 Lemnla 3.8. Let f = ft,+ fl, where f = -oE6)boço, .fJ = -oo bô(;o (3 = O. 1) := Ibl ) < U = o. ) with oi 1 and ri = 1/cj. If the "'trim and ct'" algorithm with e = eo + e (zj > O) bas b«cnappli«d to f . r«pwsent«d as obove in place qf Stop 1. then (3.23) Te .¢+,)(f) eo + and consequentlg (3.25) c(f) «,,-o + c,,-,,ç], ,, = 1.2,. u,ith c depending only on oo. t. and th« param«lcrs qf T. Pzvof. AIl the clcments for thc proof alrcady appcar in this sui»section, espccially in thc proofs of ThCOl'Cln 3.7 and Lclnlna 3.5. and wc hall assume COlni)lctc falniliarity with the llot.ation, tCl'lniliology, a.nd estilnates given thcl'C. Denote the mmd)cr of cells used in the "triln and eut'" algol'ithlii for (ho), with approximation error e, bv ri(e). Silnila.rly, let nj(ej) l»e thc Colrcsponding lmmber of cells uscd for f,i (j = 0,1), again rel)reselte, l as ]'J = 0ç), iii place of Step l. çhe theoreln will be proved once we establish the estinmte (3.26) ,(e0 + el) 2 (,,0(e0) + ,l(e) ) for anv «0, «1 > 0. Indeed. bv COlnbilfing this inequality with the results of Theo- rem 3.7 (in particular, incqualitics (3.16) (3.17)), ve can see tha.t the estimate (3.27) n(e0 + ml) 2ç«rO,Ç TM + 2ç«r1,Ç ri : is true if we set e := (ç)l/r?l--1/r, j = 0, 1, where c is the constant appearing there. But the fact that n n(e0 + e) and the dcfilfition of (-) imply TC c(f), ¢+,)(f) e0 + ci. Helwe, bv the defilfition of tlle êj, thc rightmost terres of this last ilmquality are bounded by the desired terlns on the right-hand side of ilmquality (3.25). I1 order to prove estimate (3.26). we only lmed to estilnate the lmmber of seg- lnellts for f. First observe in Step 3 of/he algorithm that for the thresholdilg condition (3.6) to hold for f, with z := e0 + e, the «onditiol must also be satisfied for that saine cell 0 ¢ for at least one of the fJ with corresponding threshold ci (j = 0.1). This shows that the tree F" = F'(f,e) determilmd bv threshold e is contained in the union of the corresl)Olding trees F'(ff, eg) (j = 0.1). By the COll- struction of segments « from maxilnal chains of F'(f) il Step 4, the Seglnents for f are disjoint and one of the conditions (3.10)-(3.12) m,,st hold. If (3.10) or (3.11) holds for a segment « of f, then 0« Ib + bl > e0 + e implies the correspond- ing condition for at least ont of f0 (and e0) or ri (and Zl). That is, for one of j = 0, 1 we must have 0« Ibl > zj, a,d s fl- at let hall of the segments of f this condit.ion must persist for a fixed index j (j = 0, 1). The lmmber of relnlmnt segments (see (3.12)), on the other hand, may be estimated bv the Stilll of the number of remirent segments of f0 and fl, plus the lltlnlber of ?lew bralcllilg cells which lna3" arise within the union of the trees of f0 and ll. These new colis are 2606 B. KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY introduced in F"(f, ) when two chains, exclusive to each of the F"(f j, ej), meet, therebv dividing the existing chains for each of the trees and creating an additional seglnent. It is easy to see that the nulnber of such new branching cells does hOt exceed nlill { rï (f°, e0), Uï (fl, gl ) }- This accounting of the three quali(ving conditions (3.10)-(3.12) for segments gives lllgX{#(f 0 0) #(fl,l)} + lllill{Fï(f 0,0),rf(fu 1 #2(f °, g0) + #(fl, gl), which iml)lies lhe desired estimate (3.26) a.nd COlupletes the proof. 3.2.2. The «ose 0 < p < oe. XX' now l'eturl to coml)leting Steps 3-5 in the ce lha.t p < oe. The arguments are quite silnilar to the case p = oe in the previous sui)section, and we shall use the notation there and indicate only the differences. hdrodu('e a new l)a.raln(,ter O, where 0 < ç < p. and fix g > 0. Step 3. TrblTniTg of (-)" (1 _< , _< K) witb e. This step is the saine as in Case 1 (p = oe) wilh (3.5) l'('l)lace(l i)y In contrast to lhe case p = ., the error 1£ - fr I1» i 1o longer controlled soMv by g. It will (h'l)end Oll the smoothlmss of the flmction f that is being approxilnated (see Theorem a.9 bclow). Step 4. Partitioning the branches of each tree F" into chains and the chains into "segments". We proceed exactlv as iii thc case p = c, repla.cing conditions (3.10)- (3.12) by the following: (3.29) (3.30) Ibo, II01 /» > e (cae of p = 0). 1/0 /i+p \ 1/0 <_ e, but y'Albo, llOll/") °) > e (case of p > 0), j=i (3.31) % IIo.,-I'/') _< . Step 5. Rewriting elements from certain segments of ". This step is exactly the saine as for the case p = Error estimation for the "trim and eut" algorithm (case 0 < p < oc). Suppose that the "trim and cut" algorithm bas been applied to a function f with 0 < c0 < p and e > O, as described above. Let ArC(f)p = eeh ceçe, A C . be the approximant produced by the algorithm. We denote (e) := #A« TC , A.(«)(f)p := IIf- AT C(f)»llv and TC TC . A,, (f)p := mf{A,(«)(f)p n(g) < 'r}. NONLINEA1R PIECEWISE POLYNOMIAL APPROXIMATION 2607 Theorem 3.9. If f G 177(7). where c > 1/0- 1/p a,,d - -- (c + )-1. then for each > O. (3.32) TC and hence Proof. We first estima.te (e). inequality of (3.28)) in Step 3, it follows that From the stopl)ing tilne critel'itm (the converse (3.34) e < ( , (IbollOI1/p)o) /° ( \ l/r (IbollOI1/P) r) (SillCe T _ O) \0C0 o 0C0'» for ea.ch 00 F Ï, whi('h ena.blcs us to repeat thc argmncnts from the proof of Theorcm 3.7 and obtain the estilnatc :/#Fï <_ c IlfllB7(-)- In going ful'thcr, we use (3.30) iii a, silnila.r fashion and the a|ove t infer as in the proof of Theorem 3.7 that (3.35) This implies the desired estimate for t(¢). ATC It rcmains to estinlate t}lç error IIf- ¢«)(f)pllp. xx first estimate lift- fllp. To this end, we group the rcmoved cells into collections of comparable 7-11ornls. We denote by E" := {0 O" F" "0 0' for a.lV O' ' Fu, O' 0} t.he set of all cells a.t which a trimmed branch sta«ts. Note that for each 0 the inequality (3.28) holds. Therefore, we ca.n partition " into disjoint collections -j, j = 1, 2 ..... L", such that =" L =" and (a.a) 0°j 0C0 for ail j = 1, 2,..., L" except possibly for j = L ", when the leftlnost inequality may fail. Hence, since the cells Dom E" has-e disjoint interiors, and recalling that IbollO, / HboçoHv, we obtail, L j=l 0¢E 0C0 ° -- 0°E 0C0 ° L kj=l 0*eE" 0C0":" L <_ c(-2P/OeP)/P=c(L")/Pe, j=l where we used the embedding inequality (2.26). To estimate L" we once again exploit the idea used in estimating :/#Fï (see (3.18) (3.19)). Since 0 < r <_ 0. we 2608 B. IARAIVANOV, P PETRUSHEV, AND R. C. SHARPLEY We use this and the fact that the collections _j m'e disjoint to obtain (3.38) Comlfining (3.37") and (3.38), we oi,tain Il.f,,- ff,, IIp _< «(e-TIl¢;ll(r))l/Pe = ""ll.ï,,ll,/lr, and honte i»v standard sulm&livitity estimates for Lp ((I < p < ,OE) we mav estimate the SlIlII (:.3.) IIf- frllp l/p* where 1'* := rein{ 1. t'}. T(, complete the proof of the the(n'em, we must estimate Ilfr - A'IIp This «liffers ri'oto our earlier arguments in the case p = . which involved the error estimate (3.15) over a ring of a segment. For any such ring R« (a ") we use instead the estimate where we used the embedding inequality (2.26). From the above, using that ail rings {R«}«e:" bave disjoint interiors, we obtain (3.4(}) lift" - A"llp -< ( ' lift,'- "11,(,,) p )/P -< ('Olldining (3.40) and (3.35) yields and hence Ilfc --'ffc(f)pllp 1]p* NONLINEAR PIECEVVISE POLYNOMIAL APPROXIMATION 2609 where p* := lnin{ 1,p}. From this and (3.39). we oltain the api)roI)riate estilnate whiçh corresponds to (3.14) of the case fin p = 4 TC f (3.41) IIf Lemlna 3.10. Let f = fo+ fl, here f = oeo boço. f = oeo ço (j = O, 1) 1)--1 u,itb j _ > ! o - p 1 (0 < 0 < P) ad r A := 1/(aj + . j = O. 1 . Furthermore, s«ppose the "trim and eut'" algovitl«m bas been applied to f, usi9 the above represetation off it place of Step l, with 0 < Tbe we bave (3.42) re - («+«(I) (3.43) u(so -[- Sl) < C S-T°JÇ ° nt- 1 "1 " ad. tberefore, Arc{t: ) < C(ll-aOJo-t-ii-cjk/'l) il = 1. o, (3.44) J P . ..... where c depend o'ul9 ot p, 0, a0. 1, atd the pa'rameters of T. Pro@ The proof is very silnilar to the proof of Theorem 3.9. and we shall onlv indi«ate the differences, using thc notation and ideas from there Those differences are in ebtilnating Fï, and L (see (a.aa na (a.as)). Fom th stopping «riterium (converse inequality to (3.28)) in Step 3. it follows that. for 0 ° F Ï, o + 1 < (IbollOI1/) OCO 1 ( )1 +c (Ibll0ll/) OCO 1/ri / )l/r°/ ) < c o (Iblloll/p) ° ÷% (Iblloll/V) 1 \OCO o \OCO o where c o := max{1.21/°-1} and we used the fart that To, T 1 <__ O" Therefore, for each 0 F Ï, at least one of 0 < C(- (]b]]o,l/P) "rO) \0CO or 1 < ¢0 (Ibll°ll/;) 1 \OCOO must hold. Denoting by FÏo and Fl the sets of all 0 ° F Ï for which the first or second inequality, respectively, holds, we obtain O°FÏ oco 2610 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY and hence (.) # _< #0 + #> _<« (0a 0 + ï). obtain silnilar (with the saine right-hmd-side quantity) for #E and L by using the saine argument. The estinm.te for #E gives the desired estilnate for } 111W use estilnates (3.37) and (3.40) iii the proof of Theorem 3.9. with e = go + e;, together with the above estilnates for #E and L , to obtain (a.46) rc A(«+«,)(f)p 5 c(eo + ml) ffoin which the desired estinmtc (3.42) fl)llows. The filml estinmte (3.44) is proved t)v sel(,cting éj = (2('/t)l/r, which I)v out rcsult (3.43), gives that n(g0+gl) n «llld Ho Ac(f)p A(eo+e)(f) p «'ll/P(gO +E1) «" II--a°,ço+ I--al.çq whore we Imve used (3A{i) in the s(,('ond in('qualit3ç 3.3. "Push the error'" algorithm. The i(lea of this algorithnl to oilr knowledge first al)I)eared iii [5]. ()lit goal is to a(laI)t this algorithnl for nonlinear n-terre ('Olll'allt ('lelllellt al)i)roxilnation iii the lllliforlll llOrlll alld perfect it so that the resulting algorithln achieves the rate of convergence of the best approximation. Iii §a.a.l, we describe the "lmsh the error'" algorithln iii its simplest and nlost naive forln. \Ve follow with three exalni)les which illustrate deficiencies of the simple algorithm and the types of traps to which it lllay fall prey. In §3.3.2, we give our refined version of that algorithm. Throughout this section. ,ve assunle that 7- = l.J=0 OE,, is an LR-triangulation of solne compact polygonal dolnain E in IR 2. where the approximation takes place (see §2.1), and f _ C(E). 3.3.1. A naive "'push the error'" al9oritbm (p = oc). We t)egin by outlining the basic clClllents of the algorithm. Step 1 (Decoropose). In this sut)section we denote by Qj(f) the piecewise linear contilmous fimction that interpolates f at the vertices I of all triangles froln . Clearly f C(E) cal1 be represented as follows: (3.47) f = Qo(f) + (Q(f) -Qj-l(f)) =: co2o. 3=1 0 where the series converges unifornlly. Ill practice the series ternfinates at some finest level a (J > 1), so that J Assulning that initially f = 0ea coço, t.here exists a fast and efficient, procedure for obtaining (3.47). Step 2 ("Threshold'" and "'push the error"). Fix e > 0. We shall begin at the coarsest level f0 and proceed consecutively through to higher resolution levels NONLINEAFI PIECEWISE POLS:NOMIAL APPFIO\IMATION 2611 O,O2 ..... (9.I- V'e define A0 as thc set of all cells 0 (90 such that [coi > E (ll011 = 1), and sel 0Ao 0o Next we rewri[e all remainilg Ierlns coco (0 A0) at [he next finer level and add [he resulting terlns to the correspolding terlns froln (coo)oeo. Thus we obtain a represelda[ion of f iii [he forlll 0 j=2 0 next process Ille CouranI elelnents at level 1. ç <lefine At as the set of all 0 #) such that IboI > e, a.nd set A := OE0eA, bobo. AI1 lelnaining terres bobo, 0 (-) At, we rewrite at the finer level and add thc resul[ing terres o the corresponding [erlns (coo)oeo:. The representa[ion of f at [his stage is written as .l (3.48) f = 1 + A1 + 00 + CoCo" Ve COllIillllP in this way until we reach the filleSt (i.e., highest resolution) level j. The only lnodificatiol at this finest level is IllaI we discard all telmS whose coecieldS in absolute value do hot exceed our Ihreshold paralneIer e. In Ihis wav Since only slnall terres (lb01 _< e) at a single (in this case, finest) level m'e discarded, they ca.lmot stack up, and we have Sollle lnodifications must be lnade, however, to insure that this silnple and efficient algorit hln will achieve sparse l'epreseltatiols iii ail asylnptot ically optilnal sense and avoid hidden t raps that will result in using too llla.ny terlllS iii the approxiilmtiolL We indicate briefly each of the possible pitfalls to keep in nlind, before developing the algorithln in fldl in the lleXt, sllbsectioll. Trap 1. The interpolation schelne we used to l'el)l'eSeld f iii (3.47) leads to diflïculties, since it does hot always lead to sparse represelt.ations. "Ve give here a univariate exmnple which lllaV be easilv extended to two diInensions. Let E := [-1, 1], and let f be the hat fimction on [ ',1 2__N_]l for N sufficiently lin'ge, i.e., f(z = ç(2Na ") with ç(x) := (1 -Jxl)ll[_l.ll(X), x IR. We assmne that 7 consists of all dyadic subintervals of [-1, 1]. Using the interpolation schelne described in Step 1 at the coarsest level, we nmst interpolate the extremes at -1,0, 1 in order to decrease the L error. The resulting error after this stage, however, is 1 - --v. Proceeding with the haire "push-the-error" algorithm with any < ½ results iii ail index set A with #A N. However, the best approximation is achieved using the single fine scale element ç(2Nx). Therefore, any reasonable algorithln that retains terres in the approximant should give a rate of COllvergellCe O( -'r) for any 3' > 0. 2612 B. KAIRAIVANOV, P. PETIRUSHEV, AND R. C. SHARPLE st Trap 2. For a given ¢ > 0 the algorithm as currently described may produce a great lmlnber of undesired terlns due to the superposition of a large number of fine level nonintersecting terlnS (coço) with a singlc coarse level terre (3.49) f=e([-1.l]+o). set a.s a set of disjoint cells 0 rioto level 2 with 0 C (-5. 5), where 5 = 2 -'. It is clear that we tan choose thesc cells for , so that «M = 2 N. At the central vertex x0 of each cell 0 we have f(.ro) > e(1 - 5) + Se = e. The "push-the-error algorithm'" will pro(luce an ilmcient apl)l'oxinm(iolL since it will hot select the ('oarse fil'St terre in (3.49) as one might hope. Instead, no such element will be ('hosen at the ('oarsest h, vel, and thc errol will be pushed. At each successive stage the co«'c'ients of the rewritt(,n des('endan Couralt elements for 00 will ail again lie I»encath thc threshold and be furIhor rewritlen until all cells are on level '2 N At that stage they will be combined with the relnaining terlns in (3.49). The corr(,sp¢mdilg ce,Ils will now bave coccieltS that exceed the threshold and nmst I)e s¢'l¢'('tcd. In'Oducing at h,ast 2 N Wrms in the api)roximant. As indicated al)ov(', a d('sirablc algoriIhln should haro anticipatcd the trap of lllanv sma.ll, finely suI)i)Olted clcmcnts Ihat may corne, ai a laie stage, and would have chosen for this flmcIion the aI)l)roximaIion (with thrcshokl e) that consists of a single elenmnt, namely e ç[-1,1l- ap 3. The final examplc is onc that outmalmuvers a quick remedy to Trap 2, i.e., meroly thresholding ail snmll terres at Ihe finest level. For a given e > 0. we defilm f= e [-.11 + Sj ç[0.z-l + eç[0,Z-MI , where mi = j2, 5j = 2-1e ' and BI = 2 N. In this example, elelnents are again building near the origin, but now aI)pear at lllany levels with small alnplitudes. The "'push-the-crror" algorithln will again take no elements at the coarsest level and push the error to the llPXt lcvel. Coll[illtling with the algorithln, we are forced to take essentiallv ail tOllllS as the approxilnation to the given fllllCtiOll when. optimally, only two terres lmed be takcn. It is obvious that we can take each of these telnplate exalnples as building blocks and i)uild flmctions to cause these problems for ail «, at ail locations and scales. 3.3.2. "'Push the error" algoritlm in the uTiiform orm (p = oc). In this section we indicate the refinements lmeded in order to gua.rantee that the "'push the error'" algorithm will achieve optimal rates of approxinmtion. As with the "'triln and eut'" algorithm, we break it down into lnanageable steps. Description of the algorithm. Step 1 (Decompose). For f E C(E) initiallv represented by (3.1), we lllcy assume, without loss of generality, that there exists a finest level )j (J > 0) such that f is written as (3.5o) NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2613 Step 2 ( "'Prune the shr'ubs"). In the current algorithm we are hot able to organize the cells of O into trees as we did in the "trim and eut" method, since, once we rewrite the error on a finer level, a.djacent trees are immediatelv affected and we lose the benefit of their intended organization properties. This step of our algorithm, however, is analogous to Step 3 of the «trim and eut" algorithln. fix e > 0 and let e* := e/2. Oto" goal is, by discarding small insignificant terres boço in the representation of f ri'oto (3.50), to prevent our refined algorithm from heing trapped by a situation such as that described in "Trap 2"' (sec the haire "Imsh the error'" algorithm of 3.3.1). b shall remove such terlns, but insure that the resulting uniform error is at most ê* and dcnote by F the set of ail retained cells. Iu addition, we shall construct a set F I C I', consisting of "final cells'" in F. First, we lmed to introduce ail organizational concept as a replacemcnt for the tree structures of 3.2. shall say (figuratively) that a cell 0 G sits on anothcr cell 0 G (-), if 0 is at least as fine as 0 and its interior (dcnotod by 0 ) intcrseçts the interior of 0% Fnrtherlnorc, tbr 0 G (-), we denote the collection of all cells that sit on 0 by (3.51) Y0* := {0 e O: 0 0 ° ¢ 0 and level(0) k lew'l(0°)}. The procedure of Step 2 will bcgin at the fincst level and proceed to the coarsest, level by level, coustructing sets F I aud I'. To initialize the procedure we Imt into F f ail significant cells 0 G Oj, i.e., such that bol > e*. place in F anv cell ri'oto Oj that sits on a cell from F The inductive step proceeds as follows. Suppose that all cells from ) with lcvels j > m (0 m < .1) have already been processed. now describe how to process ,. place into F I all cells 0 (-), that satisfv (3.52) Ibol > *, and for which there is no 0 G r I from a higher level (i.e., > m) that sits on 0*. A cell 0* rioto We mv consider the current version of r f as an intermediate (m-th) version of a final set for F. Obviously, a cell 0* from O is discarded and hot placed in F if (3.53) Ibol *, and there is no 0 The procedure is ternfinated after O0 is processed and Step 2 of the algorithm is completed. The two sets of cells F and Fy (F/ C F C (r)) produced bv Step 2 have the following properties, which follow directly from their construction: (i) if 01.02 F I and level(01) ¢- level(0.2), then 0ï ç?0_ = 0: (ii) for each 0 ° F I, the inequality (3.52) holds: (iii) for each 0 ° F, there exists 0 FI that sits on 0 °. Vre set .ff := 0er boço and define { bo, if0F, (3.54) ao := 0, if 0 6) \ F: 2614 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY then obviously (3.55) fr = Z aç. It follows fronl the construction that (3.56) [If- frll - ¢*- Indeed, to see that this estinmte holds, we let D denote the set of all cells 0 6) that were (liscar(l('(1 (luring the iml)h'lnentation of Step 2. i.e., T) = 6) \ ['. Let .r E |)e arl)itrary. If .r [,-J0ez) 0, then x does hot belong to anv cell tllat was (liscar(h,(l, and so .fr(x) = f(,r). On the other hand. if x (5 [,-J0ez)0. then tllere exists a oeil 0 (5 T tlat contains .r and has coarsest level. Since 0 ° ,cas discarded. the in('(luality (3.53) lllllst h()hl. I! follows that If(x)- fr(x)l = I Z bo?o(x)l <_ OD OEYoo wh(,r(, wc have l(,rlnaliz(',l our (,h,m(,nts s(, that ]]ç0]] = 1. This verifies the desired in('(lua]ity (3-56). Stel) 3 (Pu,s'h the error). \Ve n()w l)ro('ess cells of fr with e*, starting fronl the coars('st level (-)0 and continuing to fin('r levels. The outcome of this step will be an al)t)roxinlant A := A(f) of the fi)rm (3.57) J J 3=0 3----00GA. where A./ C (-) and Ai will det)end on f. \\e use the notation .to := {0 (5 O-0 ° çl0 ° ¢ 0 and level(0) = level(0°)} for cells fronl the saine level as 0 ° which are adjacent to it. We start fronl the representation of ff in (3.55). We define --\o as the set of all 0 Oo such that f01 > e* (ll011 -- 1), and ve set -\o := [,-J0eÀo ,U0. \Ve denote .40 := Z aoço =: Z doço. 0Ao 0A0 For each 0 ° 6)j, ç0o call be represented as a linear combination of ço'S with 0 6)j+l- \Ve use this to rewrite (represent) all renmining terres aoço, 0 6)0 \A0, at the next level and add the resulting ternls to the corresponding terres aoço, 0 6)1- We denote by do¢?o, 0 6)1, the new ternls, and therefore obtain a representation of f in tlle forln f = A0 -I- J OG j2 0 Continuing with the next level, we define / as the set of all 0 6)1 such that Idol > :, set Al := [,JoeA eo , and define ,Al :--- 0eh do¢?o. As for the previous level, we rewrite the renmining ternls doc2o, 0 (01 \ Ai, at the next level and NONLINEAR PIECEWISE POLh NOMIAL APPROXIMATION 2615 add the resulting terres to the corresponding t.erms aoço, 0 c:_ t92. We obtain the following representation of f: f = No + A + E dodo + J j=3 0E(gj We contilme in this wav tmtil we reach the highest level of cells 19j. At level 19j, we define Àj, A j, and `4.j as al}ove and discard all terres doço, 0 e_ (-)j \ Aj. We finally obtain our aI»proxinmnt `4 = `4P(f) in the form (3.57). We denote J A := A« := l J j=0 Ai and À := À := LJg0 Àg, and so `4 = Y'-0e^ doço. Since we throw away only elements doç)o with Idol <_ e* at the finest level 19j, we have the estimate Hic - `411 O(-) j \ A j and hence, using (3.56), (3.58) This completes Step 3 and with that the description of the algorithm. We want to point out an important distinction between the "push the error'" steps in the above algorithm and the "naive" algorithm described in §3.3.1. The difference is that each time we put a significant terre doço (Id01 > e*) into ,4 we also include the neighboring terres (i.e., from the index collection .Vo). This prevents our algorithm from being defeated bv a situation like that described in "Trap 3"" in §3.3.1. Error estimation for the "push the error" algorithm. Suppose "push the error" is applied to a function f with > 0. and `4(f) is the approxinant obtained: `4ff(f) := -'0e^, doço. As in the "trim and eut" method, we use the corresponding notation n(e) := #A«, P P and P(f) := P(f,T) := inf{P()(f) "n(e) <_ n}. We reinark that if f 6 B(T), then by the Embedding Theorein 2.7 it follows that f is continuous. Estimates (3.59) and (3.60). established in the following theorem, imply uniform convergence of the "'push the error'" approximants to f and provide the necessary rates of approximation bv the method. Theorem 3.11. If f c:_ BT(T ), >_ 1, r := 1/c. then for each e > O. (3.59) P A(e)(f) _< e and ri(c) < ce-HflIB7(oE), where c = 6N. Furthermore, we bave (3.60) AP(f) < c-llfllB(» . = 1,2 .... with c = (6N O). 2616 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY Pro@ In order to prove (3.59), we first observe that the direct approxilnation estimate ,&/x P ,(«)(f) <_ e follows from inequality (3.58) in the constructioll of the algorithm. Therefore it only renlains to show that #A_ _< ce-TIIfl[).(7). Clearly, (3.61) A« _< (N0 + 1)(#À), and we need only estimate the cardinality of := . We split À into two disjoint sers,/ï'I and/ï,,.. \Ve define/ï,i as the set of all final cells iii A. that is. the set of ail 0 Ç/ï, for which there is 11o 0' Ç/ï, of a higher level sitting on 0. We set .; := \Ve shall lnake l'eI)eated use of the fi»llowing simple lenmla. Lelnnla 3.12. Suppose Ad C (-) satisfies the condition that cells front different let,els do ot hat,e interiors that intersect. Then each 0 Ç (-) may sit on al most No + 1 oeils from . Pro@ The silnp]e hyp(»thcsis of thc lonlnla just states that for a cell 09. to sit on a «cil 0, it lnust be on the saine lcvel: but lhere can be at most N0 + 1 such cells. [] We first estilnalc the nuln|mr o[ clelnents Ç/ that arise as final cells in Step 2. r each 0 ° Ç 1"i, we have, by (a.52), (3.6oe) e* < Ibol ( Ibol) '/r ( ). OE'oo 0E'0o Clearly. FI satisfies the hypolhesis of Lenmm 3.12 (see Property (i) of FI, which is stated following (3.52)). and hence each 0 Ç (-) mv sit on at most + 1 cells from Fi. Using this together with (3.62). we obtain llfll(oE) := lbol ( + 1) - lbof (. + 1)-(#Fi)(e*) . 00 O'»F I 0'0o which, since "r _< 1. ilnplies (3.63) #I'i < 2(N0 + 1)e-rllf]lv(7). We next estilnate #]OEI" the lmlnber of final cells for the index t Ç constructed in Step 3. Clearly Donl that construction, a ccll 0 Ç Ç lnaV occur onlv if 0 Ç F, and hence C F. On the other hand, from Step 2, for each 0 Ç F there exists 0' Ç F sitting on 0. Therefore, for each 0 Ç I there exists 0' Ç F I sitting on 0. But satisfies the hypothesis of Lemlna a.12 (with ,M replaced bv I), and hence a cell 0 Ç F I lnW sit on at lnoSt + 1 cells froln I- Dom this and (3.63), we have To colnplete the estimate for , we must estilnate .. Suppose 0 ° Ç := I, and let 0' À be a cell sitting on 0 ° with level(0') > level(0 °) md such that level(0 ) is the minilnUln of the levels of all cells in sitting on 0 °. Such a cell exists, by the definition of À, but it is possibly hot Ulfique. XX denote by Zo the set of ail 0 Ç F which, while "pushing the error" from 0 ° in Step 3. have contributed to the terre do, ço,. Due to the lninilnality of 0'. we see that (3.65) do, = do,o,(vo,) = OE Zoo NONLINEAR PIECEV¢ISE POLYNOMIAL APPROXIMATION 2617 where t'0, is the "central vel'tex'" of 0'. using (3.65), Since 0' E À, then Ido, I > e*, and hence, (3.66) * < Ido, I <_ Il bovo,l <_ (r 1). 0Gîoo It is easilv seen that ea«ll 0 G Zo satisfies the folloving properties: (a) 0 0', (b) level(0 ) < level(0) level(O'), (c) the "central vertex" of 0 lies on 0 °, and hence 0 sits on Property (a) follows by ohserving that the support of an elelnent which is rewritten at a finer level always «ontains the supports of the COltritmting fille" elelnents. Property (b) holds, since . ç A, and hence 11o terres boço with level(0) level(0 °) lllay contribute to do, ço,. Note that it is possible that thel'e are 0 that salisfy properties (a)-(c) above but do not belong to 0.. Next. we show that each 0 G [" may belong to at lnOSt + I sets Z0* with 0* À,.. Ind«ed. let O F and sui»pose 0 ° .,. is such that 0 Zoo. In the following, we shall use the notation ri'oin al»ove that inw)lves 0 c, but we will consider su«h 0 as arbitrarv in . Let .Mo denote the set of ail 0 Ç such that 0 G Zo,. In particular, 0 ° G Mo- k, fix Mo and shov that it satisfies the hypothesis of Lemlna 3.12. Indeed. let 0, 02 G -0 from different levels. But this ilnplies 0 G Zoj (j = 1,2), and we lllV as well consider 0 = 0 ° and sav 02 = 0 , where level(0 ") ¢ level(0). Evidently. level(0 ") < level(O'), ri'oto property (b) applied to 0" and 0. Bv sylnlnetry, we nlay assume level(0 ") < level(0°). If (0) (0°) ¢ 0, thon 0 ° sits on 0 and hence, since level(0) > level(0°), 0 Callllot be iii Z0, which is a contradiction. Therefore. (0) (0°) = 0, which verifies the hypothesis of Lelnma 3.12. Now that Lelnnla 3.12 can be applied to M0, then 0 ( anv other cell froln ln<V sit on at most N0 + 1 cells 0* G «0- Therefore, 0 lnav belong to at most X + 1 such sers Zo* with 0* ,.. Using this and (3.66), we obtain I[/[IBe() lbol (So + )-1 IboV (; + 1) -l(Aï)(- OGF OO,. OGZoo Therefore. it follows (recall that ç < 1) that I combine this estilnate with (3.61) and (3.64) to obtain the desired estimate of #A« in (3.59). Estilnate (3.60) follows ilmnediatelv ri'oin (3.59). The following lemlna will be needed iii 5. Lemma 3.13. Let f= f0+fl, where f= oeoboço, f = oeobôço (j = 0.1), ,d = + (u 0 e ), d := Ib$1" < , (j = 0.1), where oo. al 1 ad r0 := l/a0, ri := 1/al. Furthermore, suppose that "'push the evror'" is applied using the above repreentatio of f , with := 0 + 1. where OGZoo 0 o 2618 B KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY £0, £1 > O. Tben we bave (3.67) Aff(o+)(f) _< eo + 1, 1(£0 q- £1) -- OEr°-A/' -° -[- C-'/'IH; 1, Therefore, #à; < (No + 1)(#r;) < (No + 1)(#Fï + #F}) (3.72) < 2(N0+ To complete the proof, we must next estilnate #Àr. For each 0 ° G /r, we define 0' G/ and Zoo exactly as in the proof of Theorem 3.11. Similarly as in (3.66), we and hence ( rj <_ I) :/#F _< 2(5o + 1 )e A.. (3.6s) wheve c = 6N. Co.nsequently. A, (f) < co-«°JV'o + CI'--alI, "11 = 1.2 .... , (3.69) P , « = Pro@ follow in the footstct)s of the proof of Theorem 3.11. shall use the notation from there, and only indicate the differences as they arise. denote e* := e + e with Q := e/2, j = 0, 1. Estimatc (3.67) is immediate from the description of the algorithm. It remains to provide estimate (3.68) for the mlmber of terres used in the ap- proximation. As iii (3.61), we have (3.70) "(g0 +g) := #A 5 (No + where we denotc À := , and ÀI and À,. bave the saine definitions, proceeding exactly in lhc i)roof of Theorcm 3.11. Çonfimling as there, we have to estimale #F I. For each o FI" we have, by (3.52) and lhe fact thal 0 < ri 1 (j = 0.1), that 0EY0o OEY0o 0EY0o ( b[r°) 1/r° +( [b[T1) l/ri" OEYoo OEYoo Dom lhis, it follows/ha.t, for ea«h 0 ° E F S, al let one of (3.n) 4 <( Ibgl) '/° or «1 < ( Iblrl) 1Ci OYoo OEYo must hold. dcnote by Fï and F} tlle sers of ail 0 ° E F S such that the respective condition from (.71) hohls fi»r either j = 0 or j = 1. For j = 0.1. we have similarly, in tire proof of Theorem 3.11. := ]b] 2 (N0 + 1) -1 ]bl 2 (N0 + 1)-l(F})(«ff , 0E 0OEF} 0EYoo NONLINEAR PIECEISE POLYNOMIAI. APPROXIMATION 2619 have or (3.74) <( Z Iblrl)llrl OEZoo nmst hold. \Ve dcnote by j,0. and ,[ tlw sers of ail 0 ,. fir whi«h (3.73) and (3.74) hokl. rcspe«tively. As in the proof of Thcorcnl 3.11. each 0 (-) may belong to at lnOSt + 1 sers Z0. 0 ,.. Thcrefore, for j = 0.1, .¢ lbDI , (No+ 1) -1 Ib$1 ' (q, + 1)-1(#)(¢) r', and hence Therefore, #X{ _< 2(No + 1)e}-rflÇ'j , j =o. 1. f _< #j,o. + #X,I. _< 2(No + 1)tero.N'oO ..11_ e-TiHïl) " This estimate, together with (3.70) and (3.72), implies (3.68) (sin«e No > 3). Esti- maie (3.69) follows by usi,lg e := (2c)",,,-"'A/} (j = 0, 1) i,1 (a.cr) a,lO (a.s) to obtain n(eo + 51) __ '/l, and so AP(f)OE _< APn(eO+el)(f)oo <__ gO + gl. [] 4. BEST ?t-TERM COURANT ELEMENT APPROXIMATION In this section, we &SSllme that 7- is a locally regular triangulation of a bounded polygonal donlain E with parameters ]V 0,/[o, r, p, 6. and :0 (sec §2.1). We denote by (I)7 the collection of all Courant elemcnts ç0 generated by 7-. Notice that is not a basis; (I)T is redundant. \Ve consider nonlinear n-terre api)roxilnation in Lv(E ) (0 < p <_ oc) from 7, where we identifv L(E) as C(E). Ollr main goal is to characterize the approxinlation spaces generated bv this approxilnation, with emphasis on the case p = oc. We let E,(7-) denote the nonlinear set. consisting of all continuous piecewise linear functions ç of the fornl ' =- Z a0990" 06.A4 where .M C 0(7-), #.M <_ n and .Ad may vary with S. the best Lp-approxinlation of f Lp(E) frolll an(f, 7)p := inf IIf- Slip. s.(7) In or(Dr to characterize the approxillation spaces generated by (a,(f,T)), we begin in this section by first proving a compalfion pair of Jackson and Bernstein 2620 B. KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY inequalities, and then follow with the usual teclmiques of interpolation of operators (see fol" exalnple [6], [15], [13]). 111 the following, we aSSlllllO iii general that 0 < p p = and a > 0 if p < : in eithcr case we set 1/r := a + 1/p. Tlleorem 4.1 (Jackson estinmte). If f B(ç). then where c dçperMs orly on a. p and [he paromelers of T. Proof. Estimate (4.1) follows ffoin anv of our constructive algoritlnns as formulated in the corresponding Th('orelnS 3.1, 3.7. 3.9. or 3.11. Theorenl 4.2 (Berlsteil estimate). If,S' E(T), then (42) IISIIB(r) w5ere c deperds only on «. p, and toc parameters of T. l'roof. shall piove estimatc (4.2) only in the case p = OE. For the proof when l' < OE, sec [11]. SllppOSe ,%" Z(ç) alld ,%' =: E0M COCO, where and # 5 n. Let A l»e thc set of ail triangles ç that are involved in ail relis 0 cM. Then N = aeA 5"5, wllerc N5 =: Ps. P a linear polynoinial. Evidently, #A V o #cM c. k s[lall utilize the natural tl'ce structurc in ç induced bv the inclusion relation: Each trimlglc En bas (COlltaills) .1[ 0 children iii En+l alld Olle parent in E,,-1, ifm k 1. Let F0 bethesetofallçsuch tllat 5D5' %rsomeS'A. We denote bv I'b the set of ail braiching triagles iii F 0 (triangles with more than one child in Ç0) and by I- the set of ail children of branching triangles in ç (which lnav or lllav not belollg to Y0)- Now, we eXtPlld F 0 to Ç := F 0 U Ç. also extend A to := A U Fb U F. hl addition, we introduce the %llowing subsets of F: the set Ff of ail filtal triatglcs in Ç (triangles iii Ç COlltaillillg 11o other triangles in I'), the set (i'0)y of the final trimlgles in F0. and the set Fch := F [k of ail chain tciatg[es. Note that each triangle 5 Fch has exactlv Olle child in F. lllaV argue as we did %r ri'ces of cells iii (3.9) that the lmmber of branching triangles does llOt exceed ttle llllnl]l,r of filial triallgles, #l'b #(F0)y, alld SillC (F0)y C A. t]lell #Çb Cil. Using tllis, we llave #F 5 6 #rb cil, #F I #A + #F ch, and # #A + #Fb + #F cl. Içeep in mind. llowever, that #ch can be nmch larger thml o. kk lleXt est.ilnate Isla() := Ee Il-la(s);, where T := 1/ (see (2.5) for the notation). We dellote, for m k 0, S,, := 0e.ee(0) Coço. shall use that. for (4.3) S(S)EE = 5(S - Sm)r IlS- .nd, lso, (S) IISII()- Recall that is the UlliOn of the collection of ail triangles froln the saine level as and which share a vertex. e denote m0 Evidently, # 3N0 #À ('* (the valence of each vertex is 5). XXb çonsider two possibilities for each & T: (a) & , or (b) & NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2621 (a) If A G H,,, then IL,, D A' for some A' G À ç T,. Using (2.3). we obtain Therefore, by summing ovcr ail m > O, we obtain in this case &EH rn_>0 AE H, m>O = cllSll # < c,llSll. l,xl-la,,(S); <_«llSlloE IAI/IzXl Sulnnfing over nz 2 0 in this case as well. we find that cllSllEE, I'1/11 OE, (.5) s «llSll s «llSll # «llSll, ,X j=u where we bave once swit.ched the ordet of summation and used that I'1 11 if ' is a child of (sec (2.2)). Çon,binin inefllities (4.) and (.5), we obt.ain ISlS, «llSll., hi«h i equiva}ent to (.2). where we used that I1',,,11,,(,) <- I1',,,11(< <_ cll*',ll(,\» <_ cllSll, applying Lemma 9.1. From the al)ove, it follows that and hence l<_a<_n,, A) E F«h rn'ir IlS- S.llL(.,» = IlS- SllL(» < cll(llSIl ÷ II/l/&)) < l.XlllSIIo, Note that if/.X_/ E "T, \ F, then Sl,, = S,,l,, and hence IlS - Srll.«,» = 0. Suppose Ai E Fc çl OE,,. For each A E Fco, we shall denote bv , (/ # A) the unique largest triangle of/ contained in A. Clearly, we have SI,x,\ ' - S,l/x¢\ ' - llA\k, "P"b and Sm]A, = llA, -P,, where P_x, is a linear polynonfial. Therefore. s(s); = s(s- s,,); <_ IlS- j=l (b) Let z_X j = 1 ..... .x, with ., _< 3N0. We have. using (4.3), 2622 B. KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY We define the approximation space A'(Lp :: Aq(Lp7 -) generated bv the n- terre Courant element approximation to be the set of ail fimetions f Ç Lu(E) such that (4.6) IIf[IA(L,) := IIfllp + k (nw«(f , T)P) q < OE' with the usual modification when q = oe. For a fixed LR-triangulation T, we dcnote bv ff(f,t) := ff(f.t:Lp.B(T)) the K-funçtional as defined in (2.30). The .]ackon and Bernstein estimates from Theorem 4.1 and Theorcm 4.2 yield (see, e.g., Theorem 3.16 of [15] and its proof) the following direct and imrse estimates: (a.7) (4.8) ct,(f, T)p < çlç(f , ,,-) + Q_l.(kcr,.(f.T)p) p'_ . p* := min{p, 1}, wh('re c d(,pcnds only on a, p. and thc paramcters of 7-. The following charaelerization of the approximation spaces 4'(Lp, T) is imme- diate from the inequalities (4.7) and (4.8), using the ot)servation (2.31): Theoreln 4.a. If o < 7 < a and 0 < q , then A;(,ç) = (,u(ç)), with equivalent norms. The next result establishes an important (continuous) embedding, which will be needed in 5 in order to identiN the approximation spaces (the ones determined by the algorithms, as well a.s best n-terre Courant element approximation) B-spaces. Theorem 4.4. Suppose out standin9 assumptios hold. i.e., a > 1 if p = OE. and a >0 ifp< . gwelet 1/r := a+ 1/p, thenA(Lp, T) cBT(T ) and (4.9) Ilfll(oE) clIfIIA(L,OE), where c depe, ds only on . p. açd the pameters of T. Proof. shall prove (4.9) only in the ce p = oe, proceeding similarly as in [7]. For a proofin the ce 0 < p < , see [3]. Suppose f Ç A(L,T), and let S Ç Z(T) be such that (.10) IIf- sll 5 oe(f. ). Since a(f.T) O, we hae f = S1 + =1(52- - S2--a) with the series con- verging uniforml% and hence (ç < 1) (.11) Ilfll() IlSlll() + II& - &-ll(). apply the Bernstein estimate from Theorem 4.2 to S - S- Ç E2.+l (T) to obtain Il&- - &-ll() «oell& - -111 5 «2((f, ). + _(f,)) NONLINEAR PIECEWISE POLYNOMIAL APPIOXIMATION 2623 and sinfilarly IISll( _< c(ll/ll ÷ Ol(f. Substituting the above in (4.11), we find that 5. APPROXIMATION SPACES FOR ALGORITHMS Our goal iii this section is to show that the algorithlns that we developed and explored in §3 acllieve (il a certain sense) the rate of convergence of the best t- terln Courant elellellt approxinlation. shall utilize the characterization of the approxilnation spaces Aq (Lp, fronl the previous section (ste TlleorelnS 4.3 and 4.4). k" shall denote bv A(Lv, T;Nr), .4 (Lv, T:Nrc), and Aï(Lv, T: N) t h« approxilnation spaccs gcn- erated bv thc "threshold", "trim and eut", and "'push the crror" algorithms, re- spectivcly. Nalnely, f Aq(Lv, T; N), whcre N is N , A rc or N , if f Lv(E ) alld with the usual lnodification wllen q = OE (it is hot quite a norm). Theorem 5.1. Let T be an LR-triazgulation of a botuded polggonal domain E 2. (a) If p = OE,, > 1, and := 1/a. tben (5.) .4(L,ç; ) =.4(L,ç: with equivalent "'norms". (b) If O < p < oe, > O. a.d := (o + l/p) -, then (5.2) .«7(L. ç: vc)= «(L. ç:v) with equivalent "orms". were "trim and cut'" is applied with parameter ç < p. A Tct ç and A(L; A) Pro@ (a) Let p = . let A(f) denote denote the approximation space generated by the corresponding algorithm. Suppose Ilfll(;) < . Eie,ltly. «,(f) (f), ,,d h,l«. ,i,lg Th,-m 4.4. It renlains to show that if ][f]] < oe, then (5.3) lfll(;) For the proof of this estimate, we shall employ Lemmas 3.8 and 3.13. Since thev are identical, it does hot matter if we prove (5.3) for "push the errof or for "trim and cut". Suppose f = ¢ beçe is the representation of f tllat is used wllile "push tlle error" or "trim and cuF' is applied. We ha.ve Ilfll := ( IV) 1/. := /, > 1. 2624 13. KAAIVANOV, P. PETUSHEV, AND . C. SHARPLEY Next, we use a well-known interpolatiol technique. We choose o0. cq. fo, and r as follows: 1 = ni < c < o0 and T0 := l/a0, T1 := 1Cl. Hence 0 < r0 < r < ri = 1. Now let (Ibo 1)=1 Se the decreasing rearrangenwnt of the sequence (Ibol)oeo, i.e., indexed so that (5.4) lb0,1 > lb021 >--- \Ve fix oe >_ 0, and dcm,te f0 := =1 boo and f := Ej=2,,+I boc2o. In going further, we apply Lemma 3.8 or Lelmna 3.13 to f = fo+ fl, froln above, to obtain 2 v +«2 - I/%1- j=2"+l Using propert.v (5.4) and the facts that r = l/c, I < c < c0, and r0 = l/a0, we infer where we used the well-known Hardy inequalities, nalnelv, we applied the inequality froln Lellnlm 3.10 in [15] to estimatc the first sure and Lelmna 3.4 from [6] to the second terre. (b) For 0 < p < c, the proof of (5.2) is Sillfilar to the proof of (5.1). The ollly diffcrence is that file approlniatc roles of Lclmnas 3.8 or 3.13 are noxv played by Lemmas 3.2 or 3.10. We omit the details. [] 6. CONCLUDING REMARKS Our t)rinmry goal iii the I)resent article is to quantify the nonlinear 1i-terlll proxilnation ri'oin Courant elements and use it to develop algorithms capable of achieving the rate of the best approxilnatiol. This is closelv related to the funda- mental question in nOlflinear approxilnation of how to lneasure the smoothlmsS of the fimctions. As we show in this article, for n-tcrln Courant elelnent approxilna- tion when the triangulation 7- is fixed, it is natural to lneasure the SllloothlleSS via the scale of the B-spaces B (7-). The use of these spaces allows one to characterize the approximatiol spaces for ally rate of convergence > 0. It also enables us to develop algorithms which attain the rate of the best approximation. It is natural to add another degree of nolflinearity to the approxilnation bv allowing the triangulation 7- to var5". Thus a flmction f should be considered smooth of order c > 0 if infT-II/llB() < ec, where the infinmll is taken over all LR-triangulations 7- (with fixed paralneters). Therefore the rate of n-terre Courant elelnent approximation to f is roughly 0(-). Sumlnarizing, our approximation schelne proceeds as follows: (i) for a given filllction f. find a triangulation 7- and NONLINEAFI PIECEWlSE POLYNOMIAL APPROXIMATION 2625 a B-space B'(Tf) in which f exhibits the lnOSt Sluoothness, (il) find an optimal represeutation of f in t.erms of Courant elements from ç, and (iii) run an algorithm that achieves the rate of the best tt-terul CHrant elelnent approximation. The first step in this schelne is the most COlnplicated one. do not have an efficient solution for this as yet. Iu the silnpler case of nonliuear approxilnation by piecewise polynonlials over dvadic partitions, this problenl, hmvever, has a conlplete and efficient solution [14]. As we show, once the triangulation ç is determined, the relnaining txvo steps are now well understood and havo efficient solutions in both theoretical and practical senses. The three algorithlns that ve develop and explore in this article provide solu- tions of the probleln under appropriate conditious. A colnmou feature of these algorithms is the first step, a nontrivial deconlposition flOlU the redundmlt collec- tion of ail Courant elellents from 7. After this initial step, however, thev take three different routes. The "threshold" algoritlnn is COnlpletely unstru«tured ]rot e3" to implelnent. The drawback of this proçedure is tlmt it is uot valid in the case of the Ulfitbrnl nornl, and as a consequeuce it does hot perform well in L v for p large. The "t.riln and cut" algorithln is wdid for L v, 0 < p OE, but it is over- structurcd and as a rcsult the tel'tbrnmnce suflk, rs. The "Imsh the error" algorithnl appears to be the preferred Ul»l»roxilna.tion nlethod. The algorithnls that we dew,lop in this m'ticle are hot restricted to t-terln Çourant elellellt approxinlation. Thev can be applied ilmnediatelv to the approxi- mat ion from (discont iuuous) piecewise al»proximat iou over lmflt i level t riaugula.t ions (for the precise setting, sec [11]). Iu this case the role of the B-spaces B7 (T) should be playcd by thc skinny B-spuces (T), introduced in (2.37). The results are sim- ilar, but sinlplify considerably. We olnit the details. Furtherlnore, these algorithnls Call easily be adapted t.o nonlinear -terln approx- imation by Slnooth piecewise polynoluial basis flulctions such those considered in [3] and, in particular, by box splilleS. The main diflçrence would be that one should use the corresponding B-spaces, developed in [3], but proceeding in a siulilar malmer to this paper. It is natural to use (wavelet or prewavelet) bases in nonlinear approxinlatiou, and specifically for approxinlation in Lp (1 < p < ). are hot aware of COlnpactly supported wavelets {prewavelets) generated bv Çourant elements or smoother piece- wise polynolnials on gelmral nlultilevel triangulations. It is clear to us that such wavelet bses vould he verv "exponsive'" to construct and hence are of linlited prac- tical value. However, iii the case of uniform triangulations, COlnpactly support.ed prewavelets and wavelet flalnes generated by Çourant elellents, or box splilleS, do exist, and have been implemented in practice. Obviously, the n-terln approxilllation fronl such bases or fralnes cannot surpa the rate of the best t-terlll Çourant (or box spline) approximation, but they may give better coustants and hence better performance results in pract.ical situations. It is also an important, observation that. even in the case of uniform triangula- tions, the B-spaces used here are different from the Besov spaces used in nonlinear approximation. For a nlore complete discussion of this issue, see [11] and [3]. Finally, we remark that in a related paper [12] we extend the argunlents of this paper to develop a corresponding approach in the Hausdorff nletric which is natural for approximating surfaces. There we also consider various practical aspects for decompositions, nmnerical approximation, and data structures. 2626 B. KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY APPENDIX. COLORING LEMMA In order to kecp focus on the main analvtical results of the paper, we bave postponed the proof of thc coloriug lcnmia used iii Section 3.2 to this appendix. This decolnpositiolI result was used to create a manageablc collection of tree structures fol" estimatiug both the error and the mmibcr of elenients used in our coustructed approxilimlit. Silice this is a gelieral purpose result which lnay prove useful in similar settiligs, we give its proof iii full in this appcndix. Fol" clarity we ha,ce broken down the proof into a scries of lelnnIas. Since the coloring is donc in several refinenient stages, itis helpful to think of thc colorilig as an ordered triple of primary, secondary, and shadc colors. Elie I)rinIary coloring b'ill sort the elements pel'iodically by l'csolution lcvel, tlic secondary colorilig will insure there is spatial color separatiolI, and the third colorilIg (shadilig) is a niore delicatc adjustmcnt to ilisure that trce structures are formcd. Wc i)egin by rcpeating thc statcment of the coh)ring lcninia for the reader's convcnience. Coloring Lemma [see Lemma 3.2]. For any LR-triangulation 7- of E, the set 6) := 0(7-) of all cells g«nerated bg T can be r«presented as a finite disjoint (0)u=l with A" = h (,oE0, «Io) (,oEç is the mimal valence and union of its subscts " " M. is the maximal tuntber of childre of a triangle in T) such that each " bas a kree structure with .respect to the inclusion lation, i.e.. if 0'. 0" 0", then ( O'F ( O"F # O. or O' c 0", or 0" c 0'. To begin the proof, we show, without loss of generality, that for the purposes of cololing we nIav assmne that the niultiresolution triangulation provides sufficient lesolution with each rcfinenient step. \Ve argue below that after a certain fixed liUlifl)er of ilicreliieltS of the levcl there will be a guaranteed refilienient of each edge and trimigle, which bv hypothesis is controlled fionl above, i.e., tllliforllllV bounded valences and nlax litllnber of subtrianglcs for each refinenlent. Consequently. we may separate the levels of O into L (L := [12NO 1112 /0]) disjoint classes (prilnary colors) by placing two levcls iii tlie Saille class iff their indices are the saine (nlod L). Thus a class is of the fornl ( 11 Uj=0 j, where (0 := Ojo for sonie 0 < j0 < L and (j := (Ojo+j L. Since each such class has a different priniary color, it will suffice to show how to dcsigna.te thc secondary colors of the nielnbers of a single . Thercfore, to sinipli" the notation aud wording of argunicnts, we will silnply refer to (secondary) coloring tlie classes ilistead of O. In Lenuna A.1 below we show however that tliese classes have additional useful pl'operties. Loosely speaking, part (a) shows that the old vertices on a given level are far apart iii terlns of the graph nietl'ic. Iii part (b) a similar statement is given for the "'central parts" of non-overlapping edges of Curant elelnents fi'oni different levels of (. For D C ]2 and m _> 0, we define the star Stm(D) inductively by çt°(D) := D and Stm(D) := U{0 G o level o = rn, 0 ° f3 St-I(D) ¢ 0}. For the vertices in resolution level m. this is just the neighborhood of radius k in the graph metric. For an edge e with vertices v and v" and au integer m > level e, we define the "central 2 R--1 ,t part" oftheedgetobest(e,m):=Stm(e\Str ({t ,v"})),whereR:=,., o +4. This selection for R has been lnade sufficielitly large so tha.t part (b) of the following leunna holds. NONLINEAR PIECEVt'ISE POLYNOMIAL APPROXIMATION 2627 Lelnlna A.1. The Courant collection ) d«scribed above sati.iïes the follou,ing con- ditions: (a) For ea«h edge [v, v'] the distance between v and v'. rneasured in the graph metric on the next finer lcvel of (--). is af least 4R. (b) If e and e are edges frein cells in ). m is an i'nteger with m - L >_ level e _> lcvcl e', and e e'. then st(e, m) ç e' = (3. Pro@ (a) Note that each cdge in (-) gcts subdivided at lcast once after 2N0 lcvels. Further, observe that al'ter / := 2\ refincmeltS of any triangle, uone of its vertices can t)c COlme('tcd te thcir oI)posite edge bv a single edge at the finer level. Using this observation repcatcdly, one ca.n verify that after L refinements, the graph nletric distance I)etween t, and v will I)e at least 2 L/l° = ]lI3 Nô > 4R. (b) Let v and v I)c the vertices of e. Using twice the observation frein the proof of part (a), we conchlde that the distance frein each of the vertices in e \ ,çtl_2No({V,v'}) to e' is at least 4 when lneasll,ed iii the graI)h nletriç on level m. Therefore. on the m-th level, e \ stl_l%({v,v'}) has a. buffe, of at least three layers of triaugles lhat separates it fl'Olll Ct. O11 the other haud. the existence of ]il0 and th« choie' of R gUa.lalit'e that 5t, ({v. v'}) D St_2N({v.v'}), and this estal)lishes the «la.iln. This COlnpletes the i)rimary colol'ing, and fi'om this point on we onlv need work with a particula.r (2) (i.e., a fixed primary color). In this case "level 0"" will now refer to the level of 0 in (2) rather than in (-). as will the star St(O) and st(e, m). Also. when referring to the color of a. ('ell we will now lnean the secondarv color, mfiess otherwise specified. For 0 ) we denote by 00 the I)oundarv of 0. and bv x0 the central point of 0. We say that the cells in -(=)' C (2),, are R-disjoint (R >_ 1) if 0 ° ç St(O') = 0 for anv 0, O' '. The next result is used for the (secondary) coloring of cells of ). proceeding from coarse to fille levels, and uses M colors, so that sanie color cells are R-disjoint. Lemma A.2. Suppose some of the cells on a 9iven level are colored in ]iI := jyô+l _]_ 1 (R >_ 1) colors se that the saine celer cells are R-disjoint. Then the test of the cells on that level can ha colored in the saine ]il colors se that the saine celer cells are R-disjoint. Pro@ Te complete the coloring Oll the given level, we first use celer #1 te paint as many cells as possible se that the saine celer cclls are R-disjoint. Next. we use celer #2 as lnuch as possible, followcd by the third and se on until either all cells get painted or we run out of colors. The latter case, however, ncver occurs. Indeed. assume the contrary and let. 0 be the first cell that cannot be colored by this algorithm with the ]il colors. The cell 0 has the property that xvitllin its R + 1 star St+l(o) there lnust be at least one cell paiuted with each of the ,'il colors. But this contradicts the fact that ]il was selected te be at least as large as the number of cells within ç/Rm+l(0 ). [] For the secondary coloring we proceed inductively, begilming at the coarsest level 0, and color cells in M colors so that sanie color cells are R-disjoint. Suppose then that all levels up t.o "-. (/," > 0) have been colored. We color . as follovs. Step a) (Color corner cells). First we define the notion of corner cell. A cell 0 of level k is called a corner cell for a coarser cell 0 if O' has ail adjacent cell 0" (a.t 2628 B. KARAIVANOV. R PETIRUSHEV, AND R. C. SHARPLEY FIGURE 2. ('orner oeils fl'Olll Step a) the smne level of course) so that a'0 lies on edge [Xo,. xo,,] and xo is adjacent to xo,, on the level /," (see Figure 2). Given a cell 0' E (2)k-1, we color each of its corner cells 0 )a. the saine color as 0'. This insures that a cell's color is propagated through all fincr levels to its corner cells. Step b) (Ex'tend the colorir.q fo R-stars of the vertices on level (k- 1)). For each vortex ' on level (k- 1), wc paint tho cells contained in çtR+2rv colors so that the coloring donc in Step a) is preserved and ea«h color is used at nlost once. This is alwavs possiile, since Al was selected sufficiently large. Note that after this ste 1) lhc saine «olor cells are R-disjoint, since part (a) of Lemnm A.1 guarantees that the stars are sufficiently separated. Step c) ( Complete the seco¢tdar'9 colorm9 of 0.). Accounting for the cells previously Iminted in Steps a) and b), we color the remaining cells from . as described in Lenmm A.2. This procedure specifies the secondary coloring of (2). and we have thus repr sented it as a finite disjoint llnion Uv=l ), where are all cells (secondarily) colored in the -th color. Thus the prilnary color skips levels until sucient refine- lnent is guaranteed, while the secondarv color insures sucient spatial separation on each level to control cell overlaps. Unfortunately. the collection of saine primary- secondary cololed cells (") lnight not fornl a tree structure, i.e., there lnight be two cells in "whose inteliors lneet but neither of theln contains the other. This lnaV only happen when a fincr cell lies on tlle edge of a given cell. To fix this defect we will set for each fixed )" the third coloring component, the hade of the cells, from two possible choices. First. we say that 0' and edge of the finer of the oeils is contained iu an edge of the coarser. X now restrict our cells to be of fixed prilnary and secondary colors (i.e.. fix and inductively determine the shade of these cells. On the coarsest level of " all cells are disjoint, and we assign them shade #1. For the induction step. we suppose cells of all levels of "up to level k have been shaded and each shaded collection satisfies the desired tree properties. We say that a cell 0 is shade-consistent with if xo does hot lie on an edge of any cell that has the saine shade as 0. Hence it is possible to place 0 in this shade collection and preserve the tree structure. In this case we will also use the terlninology that 0 is consistent with that particular shade. We now proceed to shade the cells belonging to level k, i.e., 0 , accordiug to: Case i. If 0 both touches and is shade-consistent with some coarsel" cell . then we assign to 0 the saine shade as that of the finest such 0. Recall that this finest cell is unique by the construction of . NONLINEA1R PIECEWISE POLYNOMIAL APP1ROXIMATION 2629 Case ii. Othcrwise, we assign to 0 the fil'st lllnllbered shade for vdlich 0 is COll- sistent. If no such shade exists, we introduce a new shade for (. Bv the constructiol iii the induction step. it is obviols that cach shade sub- collection has the desired tree structure. \\'c will show that these criteria intro- duce at lnost two shades. For this we need a couple of technical facts. \\'e relnind the reader that all cells bclong to a fixed )', i.e., thev havc a fixed prilnary and secondarv color. Lemma A.3. If (4 iutersects au ed.qe c' of a coarser cçll O' btt is ,ot oue qf ils Pwof. Let e' be an edge of 0' that intersects 0. and let , be a vertex of c'. Bv Step b) of out coloring procedure (for secondaly colors), St(t,) contains a ceci'ner cell 0" iii ),n that is shaded the Saille as ' . BV Step c) in the COllStlllCtiOll of St(v) does hot contain any other cells ff'oin ). Since O is hot a cOrllt,r cdl of R-1 then 0 # 0". Therefore # SI,, (t,) = , and so O nmst lll('et gl where t,' is the relnaining vertcx of e'. Th'rcforc, O C st(e', ). Lemlna A.4. Çell of )" ,ith di>rettt atades do ,ot totch. Proqf. Suppose to the contrarv that cdls 0 a, 0. " of different shades (shadc shade #k, respectively) do touch. > mav rira assume that Oj is a nmxilnal (i.e.. coarsest levcl) cell of shade #ff that touches 0t., and conversely, that 0t. is a nlaxilnal cell in shadc #k that touches Oj. This follows bv iteration and the fa.ci lhat there are onlv finitdy manv COal'ser levels; so the iteration lnUSt terlninate. lnay assume without loss of generality that lcvel Oj < lexx,1 Or. =: ttt-, and let ej.ck denote the edges of 0j.0t. respectixwly, such that ek C ej. consider the two cases under which the finer cell 0t. could bave been shaded, and show that each one leads to a contradiction. For Case i. In this event there would be a coarser cell " of shade #k that touches 0k and to which 0t. would be sha.de-consistent. Le êk be an edge of where it is touched by 0k. > consider two possible subcases, depending upon the relative level of to that of Oj. Subcase i.a k is filler than Oj. Since levd Oj 5 level t. < level 0, then by part (b) of Lelnlna A.1 either èk C e or st(êk, mk) ej = . The first possibility nmy be ruled out, since it would ilnply that the coarser cell t. would touch 0j, but 0k is the lnaxilnal such cdl of shade Hence st(êk, mk) lnllSl be disjoint fronl ey. Note that Or. is 110[ a conler cell of . If that were the case, then 0k would be disjoint ffoin the interiors of all edges on level except the edge on which «0 lies and the edges (at lllOS[ tWO, possibly one) where t- is touched by 0t.. Hence, ej lnUSt overlie one of the these edges, since it contains e. This, however, contradicts the fact that 0j touches 0k in the forlner case and contradicts the inaxilnality of 0k in the latter. Therefore 0k CallllOl be a corner cell of , and so, by Lemlna A.3. 0 c st(è«..,). But we have already proved that st(êk, mk) ej = , which is ilnpossible, silce 0k touches 0 Subcase i.b k is coarser than Oj. Since level < level 0j < level 0k, then again by part (b) of Lelnma A.1 either ej C èt. or st(e, ttz)êk = - The former case contradicts maxilnalitv of 0t. relatix to 0 a. For the latter cse, note that 0k cannot be a corner cell of Oj, because 2630 B. KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY and Oj have difforent shades. Therefore, by Lemma A.3.0,. C st(ej, mk), and so we obtain 0k êk = , which is impossible, since 0k touches on êk. For Case il. If lris case occurred for the shading of 0k, then since 0j is both coa.rser than and touches 0, 0 must hot have been shade #j consistent. Hence there lnUSt be a Oj " of shade #j that is coarser lhan 0k, and «0 belongs to some edge ê of j. X consider two possible subcases, depending upon the level of j relative to that of 0. Subcase ii.a Oj is coarser than Oj. Since level 0a level j < level 0», thon compare edges e, êj using part (b) of Lenmm A.1 to infer eilher st(ê,mt-) ej = or êj C ej. In lhe latter ce, it follows that both the edge ek (rocall Ok touches the coarser Oj on ek) and the opposite vertex 0 (since » ê) of a triangle in are contained in e, which is clear b" impossible. If the former case holds, i.e., st(ê, mk) e = , then a contradiction also results. To see this, observe that 0 cannot be a corner cell for a, due to the fact tiret they have different shades. But Lelnma A.3 implies that O C at(êj, mb-), which contradicts the fa«t that 0 e # 0. Subcase ii.b Oj is finer than Oj. Since level j < level Oj < level 0t-, we again compare edges èj. ej /lsing part (b) of Lemnm A.1 to imply either .st(ej, mg) èj = or ej C êj. By quite similar algmnent.s t.o the previous sui)case we can in'ove that contradictions are reached. Specifically, the latter statenmnt implies that both the central vertex 0» and its opi)osite edge et. belong to the edge êj. On the other hand. the fact that 0k cannot be a corner cell %r Oj will imply that Or, C st(ej, m), which will show that 0 belongs to the intersection st(ej, mk) êj, and contradict the former statement al)ove. By out smnption that different shaded cells could touch, we are led in ail ces to contradictions, thereby completing our contrapositive proof. By combilfing the previous results with the next lenmm, it follows immediatel3 that can be colored with K := 2ML colors, and the proof of the coloring lemma will be complete. Lemma A.5. Af most two shades are reqaired. Proof. Suppose in Case ii of the shading step a.bove that a third shade were needed for some cell 0. Then its central point 0 ¢ e e2 for sonle edges e of 0 and e2 of 02, where 0, 02 ¢ " are coarser than 0 and bave shade #1 and shade respectively. Now, if «0 were a vertex for e, then there would be a corner cell of 0 in " adjacent to 0. which is clearlv impossible, since cells at the saine level are H-disjoint. The saine reoning a.pplies to e2. Therefore 0 callnOt be a vertex for either e or e2, and we conclude that eï e ¢ . Hence, 0 and 02 touch, which contradicts Lemnm A.4. REFERENCES [1] C. Bennett and R. Sharpley, Interpolatzon of operators, Pure and Applied Mathematics Vol. 129, Academic Press, Inc., Boston, MA. 1988. MR 89e:46001 [2] J. Bergh and J. L6fstr6m, Interpolation spaces: An introduction, Grundlehren der Mathema- tischen Wissenschaften, No. 223. Springer-Verlag, Berlin-New York, 1976. MR 58:2349 [3] O. Davydov and P. Petrushev. Nonlinear approximation from differentiable piecewise poly- nomials, 2002. preprint. NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2631 [4] R. DeVore, 1. Daubechies, A. Cohen, and VV. Dahmen, Tree approximation and optimal encoding, Appl. Comput. Harmon. Anal. II (2001), 192-226. MR 2002g:42048 [5] R.A. DeVore, B. Jawerth, and B. Lucier, Surface compression, Computer Aided Geometric Design 9 (1992), 219-239. IkIR 93i:65029 [6] R.A. DeVore and G.G. Lorentz, Constructive Approxzmation, Grundlehren der Mathemati- schen Wissenschaften, Vol. 303, Springer-Verlag, Heidelherg, 1993. MR 95f:41001 [7] R.A. DeVore, P. Petrushev, and X. Yu, Nonlinear wavelet approximation in the space C(Rd), Pro9ress in Approximation Theory (A. A. Gonchar, E. B. Saff, eds.), Springer-Verlag, New York, 1992, pp. 261-283. IkIR 94h:410ï0 [8] R.A. DeVore and V. Popov, Interpolation of Besov spaces, Trans. Amer. Math. Soc. 305(1988), 397-414. MR 89h:46044 [9] R.A. DeVore and V. Popov, Interpolation spaces aud non-linear approximation, in Function Spaces and Applications, M. Cwikel, .I. Peetre, Y. Sagher, and H. Wallin (eds.), Springer Lecture Notes in Math. 1302. Springer-Verlag, Berlin, 1988, 191 205. MR 89d:41035 [10] M.A. Duchaineau, M. VColinsky, D.E. Sigeti, M.C. Miller, C. Aldrich, and M.B. Mineev- Weinstein, ROAMing Terrain: Real-time Optimally Adapting Meshes. Proc. IEEE Visual- ization '97, October 199ï, pp. 81-88. [11] B. Karaivanov and P. Petrushev, Nonlinear piecewise polynonlial approximation beyond Besov spaces, 2001, preprint. (http://www.math.sc.edu/ïmip/01.html). [12] B. Karaivanov, P. Petrushev and R.C. Sharpley, Algorithms for nonliuear piecewise polyno- nfial approxitnation, 2002, pr(,print. [13] P. Petrushev, Direct and converse theorems for spline and rational approximation and Besov spaces, in Function Spaces and Applications, M. Cwikel et. al. (eds), Vol. 1302 of Lecture Notes in Mathematics, Spriuger, Berlin, 1988, pp. 363-377. MR 89d:41027 [14] P. Petrushev, Multivariate n-terre rational and piecewise pol,vnomial approximation, J. Ap- prox. Theory (2003), to appear. (http://www.math.sc.edu/ïnfip/01.html) [15] P. Petrushev and V. Popov, Ratzonal approximation of real functions, Cambridge Universitv Press, 1987. MR 89i:41022 DEPARTMENT OF IOETHEMATICS, IJNIVERSIT OF SOUTH C,ROLINA. COLUMBIA. SOUTH CAR- OLNA 29208 E-mail address: karaivan@math, sc. edu DEPARTMENT OF ]IATHEMATICS, UNIVERSITY OF SOUTH CAROLINA. COLUMBIA. SOUTH CAR- OLINA 29208 E-mazl address: pencho@math, sc.edu DEPARTMENT OF ]klATHEMATICS. UNIVERSITY OF SOUTH COEROLINA. COLt MBIA. SOUTH CttR - OLINA 29208 E-mail address: sharpley@rnath, sc.edu TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETh \olume 355, Number 7, Pages 2633-26-19 S 0002-9947(03)03271-9 Article electronically published on February 27, 2003 THE ALMOST-DISJOINTNESS NUMBER MAY HAVE COUNTABLE COFINALITY .IRG BRENDLE ABSTRACT. \Ve show that it is consistent for the ahnost-disjointness lltnllber a to have couiltable cofinality. For exainple, it nlay be equal to INTRODUCTION Cardinal invarialltS of the COlltilllllllll, that is, cardimd mlmbers lmtxveen RI and c (the size of the COlltilllllllll) which atle defined a.s the smallest size of a familv of real nmnbers with a certain comhinatorial prolmrty, play an increasingly ilnportant role in moderl set theory. Equalities and inequalities between cardinal invariants have many commotions with problems arising lmturally in general topology, real analvsis and algebra, and, ri'oin a purely set theoretic point of view, there is a deep interplay with forcing theory, in particular iii the light of the search for new iteration techniques. One of the most basic questions al»out cardinal invariants is which values thev can assulne, and, for almost ail cardilmls, it is klown that anv regular value is possible, t Furthermore, most cardinals tan either be showll to be regular iii ZFC or they are equal to c in the l'andom real model, in the Çohen real lnodel. 2 or even in both, so that they can be consistently singular of uncoultable cofinality. Notable exceptions are the splitting llUlnber of which it is still ullkllown whether it mav be singular [V] and the almost-disjointlmss lmmber a. which has recentlv been shown to be consistently singular of uncoultable cofinalitv bv Shelah [$2]. Things get trickier when one considers singular cardinals of countable cofinalitv. In fact, bv far inost of the cardinals, even those singular in the Cohen o1 randoln lnodels, can be shown to have uncountable cofinality in ZFC. 4 Exceptions are Shelah [S1] has proved that the covering lmlnber of the mdl ideal may have countable cofilmlity, the ahnost -disjointness mmber a dealt with below. Received by the editors October 3, 2001. 2000 Mathematics Subject Classzfication. Prinlary 03E17; Secondary 03E35. Key words and phrases. Maximal almost-disjoint familles, almost-disjointness number, iterated forcing. Supported by Grant in-Aid for Scientific Research (C)(2)12640124, Japan Society for the Promotion of Science. lin fact, most of the cardinals that have been studied are equal to « under Martin's axiom MA. 2The models obtained by adding = + many random or Cohen reals 3Cardinals relevant for this paper will be defined below. Also see [vD], [V], or [BI]. 4This is also true for . @2003 American Mathematical Society 2633 2634 .I)RG BENDLE itis unknown whetller the reaping mmlber r [M. Problem 3.4] or the inde- pendence number i can have countable cofinality. 5 Here we show Main Theorem..4.sume CH nd let Abe singlr crdil of coq.table cofi- ,ality. Then there is a forczg extension satisfyig a = A. In particular, a = R is consistent. give a brief outline of the proof. Itis well known that, assuming C. one Call force a mad family of size R such that, by a standard isomorphism of names argument, there is no mad familv of size R for n k 2 in the generic extension [H1]. However, mad familles of size R may survive the forcing. The sinlplest wav to get rid of small mad familles is by iteratively adding dominating reals, sav for R2 steps. If this is donc in such a wav that every dominating real is dominating only over a fragment of the mad family of size Rw a¢l¢led in the initial step. the latter will survive. %t, if the dominating reals are added in a standard way. the t})rcing will lose most of its homogeneity, thc isomorphism of-names argument will ceae to work, and there may he a mad fimfily of size R2 instead. This is where Shelah's rcccnt technique of it«,ration along templates [$2] cornes in. 6 It provides for a. way of adjoining dominating reals with a forcing having enough local homogencity. However. since isomorphism-of names arguments require CH iii the ground model, we need to describe the two step extension sketched above in one step and incorporate the forcing adding the mad familv of size R into the template framework. This accounts for some of the tcchnical difficulties described below. Apart from proving the Main Theorem, we also present a new, more axiomatic. treatment of the teml)late framework (Section 1). While Shelah [$2] defines the teml)late via two procedures building more complicated sets (with larger depth) from simpler ones. we onh" require that the template is a family of subsets of the linear order underlying the iteration satisfying several axioms, most notably well-foundcdness. think this apl)roach is more lucid, apart from providing for a simpler definition of the iteration. There is a price one bas to pay for that. however: the Completeness Lemma 1.1. showing that we are indeed dealing with an iteration, requires some additional work. Of course, our general approach can also be used to prove Shelah's original results [$2] (sec [Br]) and. siuce thcre is no need to add a mad family in this case, definit.ions will l»e simpler than in Section 1. In Section 2 we describe the tcmplate used for the proof of the Main Theo- rem, and in Section 3 we provide the isomorphism of nalnes argument needed to comp}ete the proof. The template framework developed in Section 1 for Hechler forcing can in fact be used to handle a large class of easily definable ccc forcing notions (see [Br, Section 4] for a few examples). eplacing the forcing generically adjoining a mad fanfily bv ail appropriate relative, one can get analogous results for relatives of a. e.g., for as, the size of the smallest mad familv of partial flmctions from w to w. So, for example, as = Rw is consistent, and so is a = R < as = Rw. For the latter 5As for maximal almost-disjoint families, it is rather easy to force a maximal independent family of size, say, R. Both r and i are equal to ¢. and thus possibly singular, in the Chen and random models. 6As mentioned above, Shelah showed among other things that « could be singular. In his models, « is equal to ¢ and therefore has uncountable cofinality. THE ALIvIOST-DIS.IOINTNESS NUIvIBER 2635 result, one needs to replace He«hler foreing by eventually-differeut-reals forcing in the framework of Section 1 (cf. [Br, Sectiou 4]). Let us briefly recall the maill notious relevant for this paper. Two infinite subsets A and B of w are called almost-disjoint if their iuterse«tiou is filfite. A ç [w] is an almost-disjoin, t family if its lnemlmrs arc pairwise almost disjoint. A is a mad family (maximal almost-disjoint family) if it is lnaximal with respect to beiug an alluost-disjoint faufily, i.e., for everv B [w] there is A A such that A B is infinite. The almost-disjointness number a is the size of the least infinite mad fmnilv. For functious f,g w , we say that g evedually dominates f (and write f * g) if the set {n; f(n) > g(n)} is finite. The unbounding nu,nber b is the cardinality of the smallcst unbounded falnily in thc structure (w , *), that is, the size of the smallest U ç w such that for all g G w there is f G U with f * 9- The dominatin9 number 0 is the size of the least cofinal fitmily in (w , *). It is well known and easv to see that b 0 aml b a iu ZFC [vD]. He«hler for«i,,9 [H2] D (see also [B.I]) «onsists of pairs (s. f) where s G w <, f G w w and s ç f, ordered 1,y (t,g) (s, f) if t s, and g k f everywhere. It generically adds a domin.ati 9 real, that is, a real that eventuallv dominates all ground model reals. It is this forcing adjoining a dominating real which we shall use in the template fi'aluework sket«hed al»ove. Oto" notation is standard. For cardinal invariants of the continuum, we refcr to [vD], [V] or [B1]. For forcing theory, in particular for forcing related to cardinal invariants of the contimmm, see [BJ]. I thank .huis Steprgns fi»r pointing out a flag iu an earlier version of this work. 1. TEMPLATES AND ITERATIONS The lnOSt useflfi definition of a template seems to be (see also [Br]) the following. Definition (Template). A template is a pair (L,Z) such that (L, _<) is a linear order and 2- C_ 7)(L) is a family of subsets of L satisfying (1) {3, L e Z, (2) 2- is closed under finite unions and intersections, (3) if y < x beloug to L, then there is A G 2- n 7(L) such that y G A. (4) ifAG2-andxGL\A, thenAnLG2-, (5) 2- is well-founded, i.e., there is a fuuction Dp = Dp/" 2- -- On, called depth. recursively defincd by Dp(0) = 0 and Dp(A) = sup{Dp(B) + 1: B G 2- and /3cA} forA2-\{{3}. Here, Lx = {y L; y < x} is the restriction of L to x. IrA C_ L, we define ZIA = {B ç A: B Z}, the trace of 2- on A. (A, ZIA ) is also a template. Since 2- is closed uuder finite intersectious, if A Z, theu ZIA = {B Z: B C A}. In our context, we need to slightly revise this definition because we will hot only have "iteration coordiuates" LHech used for adjoining Hechler generics but also "product coordinates" Lmad used for adding a mad family. Siuce the former should be generic over some, but hot all, of the latter, we need to incorporate them into the saine telnplate fi'alnework L, and some of the above clauses should be true for all of L, while others need to be satisfied only for members of LHech- 2636 JIRG BIRENDLE Accordingly, let LHech and Lmad be disjoint sets, put L = Lnech U Lmad, and assume L is equipped with a. linear order. Further suppose 27 _C (L) satisfies, in addition to (1) and (2) above, the followillg clauses: (3') if .r G LHech and g G Lx, then there is A G Z P(L) Slch that g G A; (4') if A 6 Z and .r 6 LHech .4, then 4 L 6 Z; (5 ) the trace Z[LHech = {A LHech; A G Z} is well-founded. as well as (6) if A G Z and .r G A, then L Lma d ç A. This is the definition of "telnlflate" we shall work with for the relnainder of the imper. Notice that (5 ) lllea.llS in tmlticular that the depth functioli Dp depends only on the LHech-part, i.e., Dp(A) = 0 iff A ç Lma d and, recursively. Dp(A) = sup{Dl)(B) + 1: B G Z and B LHec h C A LHech}. (6) is a closure condition for the L,,,aa-parl which is needed to make the proof of glain Lelmna 1.1 l»clow go thlough, glore gclel'ally, we say that A ç L is «losed if A satisfies (6). (Se» Z consists «mly «»f cl»sed sets.) r arbitrary 4 Ç L, we then (Icfin« it.s «lo.ure c'(A) by c,(A) = A U .en (L Lmad)- Thus c'(A) is the slnallest closed sel conla.ining A and al(ci(A)) = d(.4). The basic idca fiw lhc following, attelnpted, dcfiniti«m con,es from [$2]. It is modifie(l, however, due te our axiolnatic treatment of the concept of "template'" (see also [Br]) and because of thc inclusion of Lm. Definition (Iterating Hcchlcr forcing and adding a lnad falnilv along a telnplate). Assulne (L,Z) is as above. X> define, for 4 Z, bv recursion on Dp(A). the partial ordcr (p.o.) P[A (more explicitly, we define P[(A. Z). but we shall drop the reference to Z in case there is no ambiguity). Dp(A) = 0. This mcans A ç Lm- P[A consists of all finite partial fulctions p with domain containcd in 4 and such that p(z) 2 for all : doln(p) for Solne u = tv w. The ordering on P[A is given by: q et.4 p if doln(q) doln(p) a.nd - ,tq , p(z) ç q(z) for all z dom(p), I{z doln(p); q(z)(i) = 1}1 1 for all i n q . Dp(A) > 0. PI4 consists of all finite partial flmctions p with domain contained in A and such that - therc is t = tv w with p(z) 2 for all z doln(p) Lmad; - letting .r = lnaX(dom(p) Lh). there is B Z P(A L) (so sP Dp(B) < Dp(A)) su«h that p[(A L) P[B and p(«) (,f) where s G w < and f is a P[B-name for an elelnent of w, such that p(A ) ,eu ". ç " (thi mlS (.) i a et-m r for a condition in Hechler forcing ). The ordcring on P[A is given by: q etn P if dom(q) doln(p) and - nq n , p(z) ç q(z) for all z doln(p) Lm. {z dom(p) Lm; q(z)(i) = 1} 1 for all i G nq " (this guarantees that the reals added in COoldinates froln Lma d are characteristic fulmtions of an allnost-disjoint falnily), well 7Note. however, that the first coordinate sPx of the condition p(x) is not a name. THE ALMOST-DISJOINTNESS NUMBER 2637 - either y = max(dom(q) ç LHech) > d:, - max(dom(p) ç LHech ) and there is B E Z ç) P(A L) such that p[(A L),q(A L) P[B and q[(A Lu) IB p[(A L), - or x = max(dom(q) LHech) = max(dom(p) LH¢ch) and there is B Z(A L) such that p[(A L),q[(A L) Net pi(,4 Læ), ],f are [B nalnes, L) Iel "](n) y(n) for all ,'" (the last two clauses mean that q[(A L)Iet "q(x) p(')"). bave hot argued yet that this recursive definition works at all. Thc point is this requires that all [A's be transitive, which is hot trivial because the sers B G Z witlmssing that q p lnav dcpcnd on the pair (p, q). Therefore, to prove transitivity, we need to show that the [B COlnpletely embed one into the other. This will be donc in Main Lelnlna 1.1 below. Note that, once this is achieved, [(A,Z) = [(A. Z[A) is imlncdiate for A Of course, the above re«ursion also defilleS [A = [(A,Z[A) for arbitrary A ç L. If A. B G Z, A C B, thon [A C [B is immcdiatc from thc definition (becausc Z[A C Z[B in this case). This is lnuch lcss «lear if one of A or B does hot »elong to Z. Neither is it clear whether [A <o [B, the lnost basiç prol)erty thc al»ove recursive dcfilition must satisfy to nmke it an iteration, ex-en in case ln»th A and B corne from Z. This issue is addressed bv the following crucial lelmna. Main Lenmm 1.1 (Colnpleteness of elnbeddilgs). Let B Z and A C B be closed. Then [B is a partial order, [A C [B and even [4 <o [B. More explicitly, any p [B bas a cammical redu«tion P0 = Po(P, A. B) [A such that (i) dom(p0) = dom(p) A (il) s ° = s for all x G doIn(p0)LHech and po(x) = p(x) for all x G doln(p0) Lmd (in particular, po = p ), and such that, whenever D Z. B,Cç D, C close& C eends Po, then there is q G [D exte'nding both qo a,d p. Note that we do not require p B P0- Pro@ By recursion induction on , silnultaneouslv for a.ll templates (L,Z), we prove that [B is indeed a p.o. (i.e., transitivity holds) for all B with Dp(B) = a; we prove [A C [B for all B G Z with Dp(B) = a and all closed A C B: we construct P0 = Po(p,A, B) satis[ving (i) and (il) for ail B G Z with Dp(B) = , all closed A C B and all p [B: we prove that for all B,D Z, B,C Ç D, C closed, with Dp(D) = and all p G [B, letting A = C B and P0 = po(P, A. B) (which has been constructed either at stage a or at an cmlier stage), there is q [D as required. Notice that if Dp(B) = a, D = B and C = A. then the latter indeed shows [A <o [B. The case a = 0 is trivial. So assulne > 0 and Dp(B) = a. first check transitivity of [B. Assume that r etB q etB P- Then clearly doln(r) doIn(p), n r n p, p(z) ç r(z) for ail z G doln(p)Lmaa, and I{z G doln(p)Lmaa; r(z)(i) = 1}l 1 for all i G n r n p. Let z = nmx(dom(r) L«ch), y = max(dom(q) LHech) and x = max(dom(p) LHh). Then there are A0, A G Z P(B L,) 2638 JRG BRENDLE such that pI(B ç Lz),q[(B ç Lz) P[Ao, qI(B c3 L) -<et,% pI(B G L), and is a P[Al-name and. in case 9 = z, ¢g is both a [A-nalne and a PA0-name as well as, in case z = 9 = z, is a PA0-name. Let 4 = 40 U Al. Then A Ç ZGç(BL) so that Dp(A) < Dp(B), and we know bv the induction hypothesis that PAi <o P[A for i = 0.1. Therefore, p(BGL),q[(BGL),r(BGL) and r[(B L) PrA q[(B L) PrA p[(B L). Moreover, is a P[A nalue and, in case y = z, ¢g is a PrA llalnP and r[( L)IbtA fg as well , in ce z = 9 = z, is a Pt.4 llalllC glld q[(B )Ikr f g. Taken together. this shows that r tB P as requiled. Now let A C B Z, A closed, bc given. Assmnc r PA. Let z = lnax(doln(r)G LHech)- Bv definition of the iteration thcl'e is A (Z[A) (Lz) such that r[(A L) Pli and is a P[A naine. There is Z[B ç Z such that Clcarly z . Bv clause (4') in the dcfinition of a tcmplate, we may therefore assume without loss of generality that B ç Lx. Thus C B and Dp() < Dp(B) = a. By induction hypothesis, PI,4 C P and [ <o P[. Therefore, 'Ç is a P[ naine as well. r G P[B follows imluediately. Hence PI 4 C required. Next assume also p PIN is given. construct P0 = Po(p.-4, N). Put z = max(doln(p) LHech)- By definition of the iteration there is Z ç(B L) suchthat=p[(BL) F[and jisaF naine. Put A= A. Note that Z[A. Bv indu«tiol hypothesis, ç has a reduction 0 = P0(ç, -, sat.is-ing the barred version of the clauses of the lnain lelnlna. The definition of p0 split.s into t.wo cases. Case 1. z 4. Then dom(p0) = doln(p) .4. p0[(A L) = 0 and po(z) = p(z) for z Ç (dom(p) A) Lx. (Note that such z must belong to Lmad-) Case 2. z e A. Then let doln(p0) = doln(p) A. P0 t(-4 L) = 0 and P0 (z) = p(z) for z (dom(p) A Lmad) L x. know by induction hypothesis that PA <o [. Therefore. there exists a caomcal projectiott o to . of the F- naine . Accordingly we let p0(c) = ,,(P More explicitly, we do the following. For silnplicity work with the cBa's A = r.o.([.) and = r.o.([) associated with . and . know by the induction hypothesis that A <o . Note that s ç . In , for all with s ç s, we let b s ç . So b = pand. for n > ] , the b, Isl = n, are a maximal antichain below . Let a be the product (intersection) of 0 and the projectio of b t.o A (recall the projection of bs to A is the unique condit.ion a such that a OE b and any extension of a in is colnpatible with b). I11 particular, a = 0 and {a; I1 = ) = 0 for > I1. Defin by recursion on n = ]s] as follows, a = a = 0 and, for n > I1, st, = ¢_. j<s(-l) a* t(_)-(j)) (which is equal to at(_ ) (a j<(_) at(_F(j)) ). Then one can show by induction on > Isl that the a, Isl = ,, are a lnaxilual antichain below P0- Therefore they canonically defiue a F-name o (that is, s ç " ' = {a, s' < s evervwhere, The main property of this naine is t.hat for ail s, a ; _ ç , I'1 = I1) is a reduction (not necessarily the projection) of bs = everywhr« ç 1'1 = I1}. (This is so because (a)' = {«-a* s' THE ALMOST-DISJOINTNESS NUMBER 2639 everywhere, sP C_ s, 18'1 = 181} is the prod,,ct of i#0 and t.he projection of b' s alld, by , ' (aS)' is trivial.) the definition of as a s This completes t.he defilfition of P0. Clauses (i) and (ii) are triviallv satisfied in each of the two cases. Now aSSulne B. D Z, B. Cç D, Cclosed, are such that Dp(D) = p P[B and P0 = P0(P, A, B). Let q0 tc P0, qo PIC. need to constru«t q. X = lnax(doln(p) LHech), , A, and P0 are as in the previous construction. Case 1. z A. So x C. Let g = lnax{z < x; z G dom(q0) can find Ê e Z[C P(Lu) such that qo[(C Lv) e F[Ê and is a F[Ê naine. Thereis Z[D ç Zsuch that Ê = C. Since.q Ê and.q follows. By clause (4 ) in the (tefinition of a telnplate, without loss of generality, ç L u. By clause (3') ill the defilfition of a telnplate filld G (Z[D)(Lx) contailfing g. Let = U U B ç L and ç = ( C) U Ê u , Ç L» By clause (2) in the defilfition of a template, G Z[D ç Z and G Z[C. Since Dpzrc(Ê) Dt)z)c(Ç) Dpz(D) < o, O,) = qo [(C Lz) P[OE 1)v the inductiol hypothesis. 0o O Po and Ç = . are ilmnediate. By the inductive assmnption for the barred version, there is q P[ extending both O0 and . define q such that dom(q) = doln(q) U doln(p) U dom(q0), n q = q(9) = q0(9) for all 9 ¢ (dom(q0) N Lz) Lnch, q(x) = p(x), qo(z) ç q(z), q(z)(i) =0, for z (dom(q0) L)L.ma and i nq n q°, p(z) ç q(z), q(z)(i) = 0. for z (dom(p) (Lz U clora(q0))) Lma and It is straightforward to check Oint q [D aud q ev q0- So let us al'gue that q ev p as well. Clearly n q n p. need to show that p(z) ç q(z) for ail z dom(p) Lma. This is obvious for z < z be«ause ç e . It is immediate by definition for z > x belonging to dom(p) dom(q0). So assume z > x, z dom(p) clora(q0). Then p(z) = po(z) ç qo(z) ç q(z), as required. Next fix i n q P. We need to check that there is at most one z dom(p) Lma with q(z)(i) = 1. By way of contradiction assume this is true for two distinct z0 < z. By construction we nmst have i n q°, x < Zl and Zl dom(q0) dom(p). Hen«e Zl A. Therefore z0 nmst belong to A as well because A is closed. Thus both z0 final Z 1 belong to clora(p0). Tlfis means that q(z)(i) = qo(z)() whi«h «ontradicts q0 etc P0, and we are done. Case 2. x A. So C. Find Z[CP(L) such that ç0 = qo[(CL) and o is a [-name. Without loss of generality, , ç C. C = is immediate. There is Z[D ç Z such that ç = C. Sin«e x , we get x . Bv (4'), without loss of generality, ç L. Ve may also qsume ç . Since Dpz( ) < Dpz(D ) = a, we can fi'eely use the induction hypothesis when dealing with A, , C, and . In parti«ular, ç0 ee P0- Now note tn,t we have s = s ç s and 0 Iee s ç f. Let m = I%1 . Since also 0 Iee "ëq >_ ] (everywhere)", we see that q0 Iete ' ff m < s eA P where we let p = Hence we get a := a qo = (the canonical reduction of ç0 to F[: note here that Dp(Ç) Dp() , so 2640 .JRG BRENDLE that/ indeed has been defined already). However. bv construction, a <-'tA/30 is nothing but a reduction of b := b',o = []m G s° p G to . So there is ri+ P[ such that + 5t ç and ri+ t b (so that, in particular, and ç+ Ikt ¢tm s). Let = p0(ç+,A. B) be the canonical reduction of ri+ to P. Then eb ç;" Therefi»re ç and 00 have a common extension ï in D[Ç. Bv inductive assmnption for the bm'red +-version. there is extending both + and XX define q such that 0 + dom(q) = dom(0 +) U dom(p) U dom(q0), v= n q() = qo() for all 9 dom(q0) LUe«h xvith 9 > x s = s and . is a P[ naine sud, that 0 + lket = max{¢, f}, q,,(z) Ç q(z), q(z)(i) = 0, for z e (clora(q0) Lx) Lmaa and i e ,,q n q°, p(z) Ç q(a), q(z)(i) = 0, for z e (dom(p) (L U dom(qo))) Lmaa and i q k P- To sec tiret q [D. note that + ItD s ç J: by construction. It is then sraightfi»rward check that q tD qu.P- In fa«t, for q ?tD P we argue in Case 1 al,ove. Note that. as an immediate consequence of lain Lemma 1.1, we get that for arbitrary closed A ç B ç L, F[(A. Z[A) completely embeds into F[(B. Z[B). Lemma 1.2 (Chain condition). Let .4 . Ang mcomdable K Ç F[A bas an un«oudable certered subsct. Pro@ Bv a standard A system argument, it suces to show that n p = t. sç = s for all x dom(p) dom(q) Lunch, and p(x) = q(x) for all z dom(p) dom(q) Lmad. then there is a common extension r with dom(r) = dom(p) U dom(q), n = n = n . if x dom(p) C/LHech, if x C dom(q) çl LHech raid r(«) = { p(.r) if « C dom(p) ç/Lnd, q(x) if x dom(q) ç/Lma d. We do this bv induction on Dp(A). The case Dp(A) = 0 is trivial. So assume DI)(A) > 0. First assmne x mmx(dom(p) Lch) < y = max(dom(q) LHch). Then there is B Z P(A L) such that pL, q[L u PB. and is a P[B naine. By the induction hypothesis, ve get the required fiL u %et pL.q[L. Let r(y) = q(y) and let r(z) = p(:) for z e dom(p)Lmad, z > y. and r(z) = q(z) for z e (dom(q) dom(p)) Lmad, Next assmne x = max(dom(p) LHech) = max(dom(q) LHch). Again there is B e Z P(A Lx) such that p ILs. q tLx e P tB, and ]. ] are P tB-names. Again we get rtL«. Let s; = s = s and ]" be such that r[L It ]" = max{j,]}. Also let r(z) = p(z) for z e dom(p) Lmd, z > g. and r(z) = q(z) for z (doln(q) k doln(p)) Lmad, Z > g. Lemma 1.3 (Embedding Hechler forcing). Let x LH«h and A T5en t5e two-step iteration FA . that canonically adds a Hec5ler-generic coordinate 3" over te generic extension via F[A contpletely embeds into FL. THE ALMOST-DISJOINTNESS NUMBER 2641 Pro@ Let B = cl(A U {x}). I?[B embeds into P[L bv Main Lemlna 1.1. So it suftïces to show P[A * b <o F[B. This does hot follow from (file statelnent of) Lenmla 1.1 because A U {x} need hot be closed, but it is relatively straightforward from the proof of 1.1. More explicitly, given p [B, there is Z[B (L) such that fi = pILe P[ and ] is a P[ llalllç. Without loss of generality, ,4 ç . By 1.1, P[A o P[B. Therefol-e, p h a CmlOlfical l'eduction ç0 [A. As in Case 2 of file proof of 1.1. there is a canonical projection ]Ço to P[A ,f ]. Define l'o e L4 * by po[A = P0 and p0(z) = (s, ]o). As in Case 2 of the proof of 1.1. argue that any q0 P[A * b extemting ivo is compatible with p. This may badly rail in case A Z because then [A * b lmed hot elnbed into Lemma 1.4 (Names fi)r reats). Assume iv [L avd ¢ es a [L ,ame for a real. Then there is A Ç L tout, table such lhal. Icltin 9 B = ci(1), iv [B, ad ] is a P[ B vaine. Pro@ The proof proceeds bv sinmltaneous inductiol on Dp(L). ithout loss of gelmrality. Dp(L) > 0. Assume first p [L. Let x = max(dom(p) LHech)- There is C Z such that p[Lx [C and ] is a [C-lmme. Bv ilduction hypothesis, there is A0 ç C COUltable such that pile [c1(,4) and ]ç is a. [cl(A0) lmme. Then p [B where B = ci(A), A = Ao U dom(p). Assume llOW that ] is a [L naine. By cCC-lmSS (Lemma 1.2), there are i, n } Ç [L and {kn,i : i. n } such that {p,i; i } is a nlaximal alltichail in [L for all n w. By the previous paragraph, we can find countal)le sers A,i such that p,i [cl(A,.i). Put A = i. 4,.i, B = ci(A). Since [B <o [L (Lelmna 1.1). we can construe ] as a [B-nalne. Assume (L.Z) and (L. `7) are telnplates and Z C_ `7. \Ve sav that Z is cofinal in `7 if for all x 6 LHech and all _4 6 `7 A/)(L:) there is t3 Zç7a(Lx) containing ,4. The following is. iii a sense, a trivialitv. Lemma 1.5 (Cofinal subtenlplates). If Z is «ofirml in . then P[(L,Z) is forcing equivalent to P['(L,,7). Pro@ By induction on Dp(L) (in the sense of 2-), we argue that conditions in PI(L,:/) and conditions in PI(L,) can be canonicalh identified so as to vield forcing equivalence. Without loss of generality, Dp(L) > O. Let p PI(L,). Put x = max(doln(p) ç? LHech). There is A C P(L,) such that p = p[Lx ¢ P[(A.,7) alld ]P is a P[(A. ,,ï)-llallle. Since 7 is cofinal iii ,7, there is B 2- ç/)(L) such that A _C B. Bv Main Lemlna 1.1. we know that PI(A, `7) <o ]PI(B, ,7) and, by the induction hypothesis, PI(B, .7) and P[(B.Z) are foI'cing equivalent. Therefore, we lllay construe /3 as a condition in P['(B,Z) and ff as a PI(B, Z) naine. Thus iv ¢ P[(L. Z). It is straightforward to verifv that this identification induces forcing eqnivalence. [] 2642 Jt:IG Bt:IENDLE Proposition 1.6 (Adjoining a scale). Assume tt is regular uncountable, p C_ LHech is cofinal izt L, and La Z for all c < p. Then I?[L forces b = = p (i.e., there is a p scale). Pro@ For each a < t I, let J: be the lmme for the Hechler-generic adjoined in coordinate a of the iteration (see Lemma 1.3). By construction, the fa are forced te be wellordered by <_*. Let ) be a I?IL-name for a reM. Bv Lemma 1.4, there is A C_ L counta.lle such that ) is a I?[cl(A)-name. Since t is regular uncountable and cofinal in L, there is a < tl such that ci(A) _C La. Since La 27, fa is forced te dominate the reals i the generic extension via II[La and, a fortiori, it will dominate .¢ [] Proposition 1.7 (Adjoining a mad family). Assume L bas uncounlable cofinalitl and Lmad i8 c&nal it L. Then FIL cano,tically adjoins a mad family of size [Lmadl- Pro@ Let G be F[L-gcncric over the ground model. For x G Lmad define t = {tf e ; p(x)(t,) = 1 for seine p e G}. Let M = {};" .r e Lmad}- By definition of the p.o., M is an ahnost-disjoint fanfily. We need te check maximality. Se let ) be a FIL naine for an infinite mil)set of w and assmne bv way of contradiction that p forces that 2 is ahnost disjoint frein ail f. By Lemma 1.4 there is a countable set A such that p F[d(A) and 2 is a F[d(A)-name. Since L has mwountable cofinality and Lmd is cofinal in L, thcre is " G Lmad such that al(A) Ç L» By Main Lemma 1.I we know that P[cl(A) <o P[L <o P[L. Find k0 and P0 etz P such that Put P0 = po[L. Clearly any , 9 dom(0) Lmd, is a PL-name. Se we tan find k k k0 and Pi tz 0 such that for all .q dom(ç0) Lmd- Since 2 is forced te be infinite, we cau find and i0 k k such that p Ietz, i0 2. Without loss of generalitv, n > i0. Then we nmst necessarily have (9)(io) = 0 for ail .q dom(p0) Lmd. Define a condition p by * dom(p) = dom(ç) U dom(p0), * p(z) = po(z) for ail z 6 LH«h * p(z) D p0(z), p(z)(i) = 0 for all i with Lmad dom(p0), z > , { 1 ifi=i0, p(z)(i) = 0 for all i It is straightforward te verify that p P[L and that p Nelz P0. Since we have a contradiction. [] THE ALMOST-DISJOINTNESS NUMBER 2643 . BU1LDING A TEMPLATE FOR ADJOIN1NG A MAD FAMILY For simplicity asstlme CH for the relnainder of the pat)er. Assulne A0 k R is regular, and A > A0 is a singular cardinal of countable cofinality, say A = Un An, the Anbeing regular, equal to A n , and strictly increasing. Also sui)pose no < A for n < A. As usual, It* denotes (a disjoint copy of) with the reverse ordering. Elelnents of i t will be called positive, and lnembers of are e9ative. For each n choose a 1)al'titiol A, = Ua<w, s,î such that each S is co-initial in ,. Also aSSUlne S, A = S,c for m < n. The following definition is lnotivated by Shelah's work [S2]. Definition (Template for adjoining a mari falnily). Define L = L(A) as follows. Elements of L m'e lOnelnpty finite sequen«es .r (i.e., dom(z) Ç w) such that z(O) x(n) Ç ; U , for 0 < ,t < Ixl- 1, and in case Ixl 2, if .r(Ixl- 2) is positive, then x(Ixl- 1) Ç AleI_ 1 u A, and if .411 - ) i negative, then .r(I.r I - l) Ç A* U Sgy .F e LHech if Ixl = or (11 - ) e * Ixl-1 U lxl-l" Otherwise .r Ç Lmad- (This lneans that x e Lmd iff Ixl 2 oe a.d either x(x-2) is 1,ositive and x(Ix- 1) or .r([x- 2) is negative and '(Ixl- 1) < * - Ixl-1-) Equip L with the following lexicographic-like ordering: .r < y iff either .rC y and (Ixl) i positive, or c . lld x(lyl)is negative, or. letting ,, := nfin{m .r(m) y(m)}, either .r(n) ia negative and y(n) is positive, or both are positive and x(n) < y(n), or both are negative and .r(,) <a. y(n) (i.e., there are a < fl < such that x(**) = ff* <a. a* = ()). It is immediate that this is indeed a linear ordering. We identify sequences of length one with their ranges so that 0 is a cofinal subset of L. Sav x Ç LHech is relevant if Ixl 2 3 is odd..r(n) is negative for odd n and positive for even n, x(xI- 1) < w, and whenever n < m are even such that x(n),a'(m) < 1, then there are fl < a such that .r(n- 1) Ç S_ 1 and x(m- 1) e S_. For relevant x, oet & = [t(Ixl- 1),x), the interval of nodes between .ri(Ix [ - 1) and x in the order of L. Notice that if x < y are relevant, then either ,J Ju = or & C ,lu (in which case we also bave Il Ixl, :F(lyl- 1) = yF(M- 1) and .r(ly]- 1) y(lyl- 1)). Define Z = Z() to be the collection of all finite unions of sets of the fornl LforaGA0, ci(&) for relevant x, cl({x}) for x Ç LH«h, and L Lad for x Ç LHch. So L() is a subtree of (* U) < (i.e., it is closed under taking iifitial segments). The nodes belonging to Lmad are exactly the terlnilml (= maximal) nodes of this tree. The point of the Jx is that we need "copies" of the large supports given by the L for isomorphismf-names arguInents. The S, then, are used to code the places where we put the .J so that we basically get well-foundedness for free. Lemma 2.1. (L,Z) is a ternplate. 2644 j01ï{G BIRENDLE Proof. Clauses (1) and (6) in the definition of template are immediate, as is closure undcr finite rotions. To see closure under finite iutersections, it suffices to argue that the intersectiou of any two sets of the above form (i.e., Le, ci(Je), d({x}), and L Ld) is again of this form. This, however, is straightforward so that (2) holds as well. To prove (3'), lct x G L«h and g G L,. In case g G LH«h, we have g G c({g}) For (4), it suffices again to consider sets A from Z of the above form. Let x G L«h 4. Without loss ofgencrality. AL« ¢ 0. If 4 is ofthe form L d(Ju), cl({g}) or L u Ld, then we must have V > x. A = L u is impossible and if A = cl(Ju), then x < g[(]g]- 1) = min(Ju). So, in each of the possible cases, the intersection with L, is L, Lmd- are left with showing well-foundedness (5). Assume A,,., n G , is a de- crcasing chain ri'oto Z[LH«h. Let a be such that Le, LH«h occurs in A, as a component. Choose a, o milfimal among thc a. Without loss of generality, n0 = 0. Then all Le, Lch are thc saine and it suffices to consider the J«-components. Thus we may assume, without loss of generality, that A0 = LH«h, and there is a finitely i)ran«hing tree T ç w < such that A,, = («¢Tm- "]x U Fn) LHe«h where the #Ç ç LHch are finite, and such that a Ç , la I = < I1 = m. implies .1 Ç ,1, and such that the .1«, a G T , are pairwise disjoint. Now note tlmt if f G [T] is a bra.nch, t/mn the sequence {.rI"; ,, G w} must eventually sta- n} wouhI constitute a decreasing sequence of ordinals. Then notice that if Ix{ is eventually constant, so is the decreasing sequence x{I'(]x I] - 1).) Since T is a finitely branching tree this means that the total number of the x is finite which in turn entails that the sequence of the .4, eventuallv stabilizes. Corollary 2.2 (Bounds for a). FIL forces b = 0 = Ao and adjoins a mad familv of size (so that o a ). Proof. This is immediate bv construction of FIL and Propositions 1.6 and 1.7. 3. KILLING MAD FAMILLES USING TEMPLATES We are left with showing there is no mad familv of size legs than A in the generic extension. As explained in the Introduction. this is an (albeit sophisticated) isomorphism of-names argument. Isomorphisms of names canonicallv boil down to certain brands of partial isomorphisms between subsets of L. and we begin with their investigation. Definition (Isomorphism). Let A.B C_ L be countable trees, s Call A and B isomorphic (.4 B) iff there is a bijection ¢ = CA,U" A --, B such that (a) I(«)1 = I1, (b) 4(«)In = (c) « < y iff re(x) < re(g), (d) x(n) is positive iff ¢(x)(n) is positive, (e) x G Ld iff ¢(x) G Ld for all x, g G A and all G w, and such that 8F{ecall A is a tree if it is closed under taking initial segments, i.e., given x Ç A. we have x[nÇ,4 for ail n Çw. THE ALMOST-DISJOINTNESS NUMBER 2645 (f) Z[A is nml)pcd to TIB via Since the trace of Z on each comltable set is comltable, there are at most 2 ° = isoinorphism types. This, the strongest notion of "isomorphism" we shall consider, vill be used in several pruning arguments below. However. tbr most purposes the following is sutiicient. Definition (Veak isomorphism). Let A./3 C_ L be a.r|fitrary. \Ve sav that .4 and /3 are weakly isomorphic (.4 wk /3) if (e) is satisfied and instead of clauses (c), (f) we have (c') x < y iff ff(x) < ff(y) fol" all .r.y sueh that there is z G LHhçA with x<_z<_y, alld (ff) ff lllaps a cofinal SIlbSCt of Z[A to a cofilla.1 subsct of respectively. Lemma 3.1. Let 4 md t3 b« countable trees such that LHch ç A (L«ch t3, respectiv«l.q) i,s cofinal i, A (in t3, resp.). If A - 13. as witnessed by ff), then there is l, ectending ff and wit.nessing that ci(A) wk ci(B). Pro@ Call a nonempty X C ci(A) V1 Lmad counected if given x < y ffoto X, the interval Ix, y] is disjoint ffoto A ç LHch. A maximal coimected set is called a connected comportent. Note every commcted COlnponent has size A (because LHech is cofilml in 4) and ci(A) ç? L,nad is a disjoint union of at lnost comtablv man.v connected components. Given x LHechç?A with .r([xl-1) being positive, put Comp. r z for every z A ç? LHech with z D x and y > z for every z A ç? LHech such that z[[x I < x}. Clearly Comp« is a commcted colnponent. Duall.v, define Comp.,. for x LHech ç? A with negative x([z I - 1) by interchanging < and >. For each y ci(A) ç Lmad. there is x LHech f-I A with y Comp. To see this, let n < [y[ be lnaxilnal such that y[n A. Assume. without loss of generality, is positive. Let k <_ '« be minimal such that all y(i) for k _< i _< n are positive. If possible choose m, k _< m < ,, and x A çl LHech , Ix[ ; .l ÷ 1, such that xlrn = ylm, x(m) > y(m) is lninimal, and such that m is the maxinml value for which such an x can be round. Then y Comp«. If m and x camlot be round, we let x = ylk and check y Comp.Æ (note that (Ixl- 1) ; .v(- 1) is negative in this case so that the second alternative of the definition of Comp applies). Therefore ci(A) Ç1 L,nad = UzLHechA Cmp.. Also notice that for x.x LHech çl A, if Comp« = Comp,, then Comp¢(x ) = Comp¢(x,), and if Comp« ç Comp, = 0, then Comp¢(x So we can simply extend ff to g, by mapping Comp. to Comp¢(x ) for all x LHech A. Then (c ) and (e) are ilnlnediate. To see (f), note that bv defiifition of the template, sers in Z[cI(A) that are Ulfions of sers ff'oto Z[A and of sers of the form L V1Lmad are cofinal in :/TIcl(A ). However, since ff identifies sers of ZIA and sets of Z[B. b identifies sers of the latter kind. [] Note that we did hot use the full st.rength of out notion of isolnorphism in the above proof. Clauses (c) and (f) could be replaced by (d) and (f) respectively. Furtherlnore, instead of dealing with tlees A and/3 (and having (a), (b), and (d)), 2646 .IFIG BIRENDLE it suffices that ci(A) çl Lmad is the ullion of the COlni)onents Comp., x E LHech A A. and similarlv for/3, and that extending ¢ by lnapping Cmp. to Comp¢(z) preserves Lemma 3.2. If A w«k B. the,, PI,4 Pro@ Notice that clauses (c), (e) a.nd (f) are enough to guarantee that D[B. Bv Lelmna 1.5, this is still true if (f) is replaced bv (g). Finally, bv the cas" PI,4 is defined recursively, interchalging elelnents of Lmad that belong to the saine COlmected COlnponent of 4 Lmad does hot affect the p.o. 9 (because the interchalging mai) sends a cofilm.l subset of Z[A to a cofinal subset of Z[A. see 1.5). Completion of ttte proof qf the kIaiz Tteorem. Now assume À is a naine for an ahnost-disjoint familv of size < A, say À is listed a.s {,4; a < h'}. Also assulne is forced to have size at least A0. Let k < be maximal such that , Ai.. Without loss of generality, h A-. 2. 2, shall perform several standard pruning argulnents, reordel'ing the family of the À' so that the first A many look very "silnilar", that is, those that do llt fit the t)att«ll get relnoved to higher indices. This is whv we stilmla.te h A- 2. Eventually. the first lllallV Àa will suffice, and it is those that we use to create a lleW llallle ,h «itnessing non-lnaximality. For fixed a, fiud c«mntablc maximal anti«hains {P,i; i G } ç tlid {,i e 9. i, , e w'} Sllch that 4" Pmi lb . iff .i = 1 and iffk.i = 0. Let B = 0{dom(p.i): i.n } ç L. B is at most countable. Without loss of generality, it is a tïe. Let C = cl(B). Put B := B . So < + < A. Bv CH and the systeln lemlna we inay SUlne, without loss of generality, that tho {; o < N} forln a -systm. ald that th bijction = ' (see above) sending B fo B is an isolnorphisln fixing the roof R of the svstem. Beca.use t here are only Aï i = - i mmv COllt able subset s of L h (* U A) . we may also assume that if' Lech and '. then z . Also stipulate that thre is SOlne 00 < 'i such that whenevr o < Nh, , j odd md (j) then (j) N for SOlne 0 < 00. As xplaind above, cmonically inducs a isolnorphisln = @' betwe«n C and C (Lemlna g.l), which in turn yiIds an isolnorphism ç = i a, bet.ween PC md PIC (Lelmna g.2) both of which elnbed iuto FL (Main Lelmna i.I), as well as between FC-imlnS ld [C-naluS. Furthernore, since connected components are holnogeneous froln the forcing point of view, since C L«h = B L«h is countable, and since C Lmad h only countably inany COlmeCt.d components (see th proofs of Lemlnas 3.1 and g.2), it bas, up fo isomorphisln, only 2 ° = N inanv isolnorphisln types of nalneS. (Of course, there are a total of A ° nalneS.) Therefore, we lllV g]so suppose that identifies with .. which in,ans, more xplietly. tlmt ,i := '.i = ' md p XçS'ite B = {z: s T} where T ç ( Ul) < is the canonical tree isolnorphic to anv B . This means in particular that '(z) = «, that I1 = I1, and that s(z) is positive iff z(n) is positive. Let THech = {$ T: LHeeh} and Tmd = {s T; z Lm }. Tn«h is a subtree of T, while Tmd is a set of terlninal nodes of T. Furtherlnore, let S ç T be the subtree of T corresponding to the root, 9That is, the order structure on connected components of .4f3 Lma d is irrelevant, and connected components are homogeneous from the forcing point of view. THE ALMOST-DISJOINTNESS NUIVIBER 2647 a a xs iff s E .9. Furthermore, that is, sE Siffx» G R for allo. So. foro ¢/3, m s = if s G THech S. then [s I OE k + 1. List {t G T S; t[([t[- 1) G S} = {t.; n k 1}. For a < fl define { mill{n; ei{her t.([t,,[- 1) G w and x a X F({,}) or t(l/l- 1) w alld a'a - - = t.(If. I 1) <a-.(It I 1)} if such ail it exists, 0 otlwrwise. Note tha.t for each n k 1, eVely subset of Ai. honlogeneOllS in color , lll/ISt be finite. Using partition calculus as well standard pruning argtllnelltS, we nlay therefore assmne that for all a < Wl, if s S and s{Ç) S. then if ç is positive, then -(ç)(s]) for all < , and all x are larger than Wl, and if Ç < , s{Ç),s{) S, then - either for ail a, fi, we have (this is t.he case when sup z(ç)(s) < sup ()(lsl)), -- or for ail ( < x.<>(Isl) < <ç>(Isl) (this is tllc cse whPn sllpa if Ç is llegative, then x-(ç)(sl) > x S'(Ç),S'{} ,ç, t]len - either for ail a, - or for ail a < , we have x a Define x s L by recursion on the length of s T follows. If s S, then x s = x s (a fortiori Ix;] = Ixl = Isl). If s S lld s'<0 Ha S, 1Pt x-«>(l*l) be th linfit of the x-<ç>(ll) (so it is either the sup or the inf, depending on whether Ç is . 0o positive or negative). Next find % < ll+l, > Wl and s SII+I, such that for all such s and Ç, if xç.<o(Isl) = sup.x.<o(Isl), thon for ail if x2.<o(Isl) = iIlf. Xs(ç)(Isl), then for -«> r(Isl + 1), we hv y(Isl + 1) < %. It is clear that we Call find such s's because ls[+l > [BI is regular (since [si k k). To complete the definition of xs-(ç ) stipulate I s-/01 = Ix-<ç>l + o I*l + 3, and define {x ,-(ç)([s[) if [si > 0, a--{ç)(Is I + 2) = ç if Isl = 0. If s S and s'(ç) Tmd S, find 7 A* OA such tiret () Lmd and for ail B with [IsI = ., we have (Isl) # . Such clearly exists because A > IBI . Stipulate Ia.-{ç)l = I .-{ç)l = Isl + 1 and let a.-{ç)(Isl) = 7- Finallv. for the l'emaining t T, stipulate again laîl = laPl + 2 = Itl + 2. find s c t with s S maximal, and ptt aî[(Is ] + a) = a and a'î( + 2) = m() for > Is I. Let B ={.'., s T}. Notice t, hat B , although verv trelike, is not a tree like the B's. We proceed to verify that (B.gB ) and (B.g[B), a < w. are weakly isomorphic (clearly (a), (b) and (d) will rail but this is hot relevant for us). 2648 .IIRG B1RENDLE a ' \Vithout loss of generalitv, o = O. (c') Fix o and define ¢ = ¢" by ¢(a ) = x. . and (e) are ilnmediate by construction. So let us check that the trace of on B ° is lnapped to a cofilml subset of [B . First fix and consider L. Notice that there .o L is ff0 5 such that ç(L&B °) = L&B = LB . For any s T with . (0) > vet x s ¢ LB one lnust have a (0) > /5 0 and x(0) is an oe-lilnit of the x(0)'s. In particular, for ail such s. x(0) nmst have the saine value, say 70- Also .r(1) = 70 and «(2) = s(0) < w. This lneans, however, that L3 B ° is lnapped to (LBO cl(a«)) B via ç whore ].ri : 3, x(0) : 70, x(1) : 7) and x(2) : sllp{.s(0) + 1; x(0) < Ç}. Next assume a" is relevant and considor 0 «r(ll-). Assulne &B ° ¢ . Then there is s T such that [s = Ix I- 1 and x s = " r ° and ,le B ° is mapped to ,& B via ç because In case s S, we have ms = s 0 l,a v(ll ) < (11- ). z e n for l,y e g0 with I1 = *, r(ll- ) = « - - [11 Ce 8 T S, let j0 ( 18] lle lnaxi,nal with sU0 e S. Defilm g by g[([ - 1) : d s "n and g(]l - 1) : x(Ix ] - 1) and note that .1«. B ° gets mapped to u B provided wc can show is relewmt. In case jo > 0 there is nothing to show lmcause whelmver 0 =*(3) > w. and. a'(3) > wt. J j0 even, tht'n also .r s(j + 2) 0 - . " " (j0) > w while, if J0 is odd. if j0 is even. we additiolmllv have a s (30) = SUl)a xs we additionally lmve s(3o + 1) = 7sjo > w. In case j0 = 0 this is true because .,'(1) Sï and 00 is larger than ail the 0 for which a'g(j) S where j > 1 is oaa. Even though B is not a ri'ce, xve can verify, as in the proof of Lelnlna 3.1. that. letting Ç := cl(B), Ç Lmad = ,eT Comp, and that ç = ' : Ça + C, et < w, which extends ç and lnaps Comp« to Comp, is a weak isonorphisln. B)- Lelnlna 3.2, ?[(' and F[C are isomorphic bv a map = '. X sends [Ca-lmlnes to [C-nalneS, and we defilm .n to be the image of .a under Bx- constructioll, it is then also ilnllmdiate that whenever < , we can find o < Wl such that B U B and B a U 3 are weakly isolnorphic via the lnapping fixing nodes of B and sending the x to the corlespondilg x, and such that this lnapping identifies cofinal subsets of the traces of Z on the two sets. m Again. this weak isomorphism canolficallv extends to a weak isomorphisln of C U Ç and C a U C, which in turn means that F[C U C and F[Ca U C are isolnorphic (Lemlna 3.2) bv a mapping sending the llalne .4 to a. Since À and . are forced to be ahnost disjoint (by F[Ca U C), so are and . ' (by the iSOlnorphic [C U Cç). Since [C U C embeds into [L (Lelnlna 1.1), this completes the proof of the non nmxinmlity of and, by Çorollary 2.2. of the Main Theoreln. [:{ EFEIENCES [BJ] T. Bartoszyfiski and H. Judah, Set Theory. On the structure of the real line, A Ix Peters, \Vellesley, MA. 1995. MR 96k:03002 [B1] A. Blass, Combinatorial cardinal characteristics of the continuum, in: Handbook of Set Theory (A. Kanamori et al., eds.), to appear. [Br] J. Brendle, Mad families and iteration theory, in: Logic and Algebra (. Zhang, ed.), Contemp. Math. 302 (2002), Amer. Math. Soc., Providence, RI. 1-31. [H1] S. Hechler, Short complete nested sequences in N \ N and small maximal almost-disjoint familles, General Topology and Appl. 2 (1972), 139-149. MR 46:7028 [H2] S. Hechler, On the existence of certain cofinal subsets of c0 °. in: Axiomatic Set Theory (T. Jech, ed.), Proc. Sympos. Pure Math. 13 (1974), 155-173. MR 50:12716 [MI A. Millet, Amie Miller's problem list, in: Set Theory of the Reals (H. Judah. ed.), Israel Math. Conf. Proc. 6 (1993). 645-654. MR 94m:03073 10In fact. this is true for ail but countably nlany 0 < Wl. THE ALMOST-DIS.IOINTNESS NUMBEIR 2649 [SI] S. Shelah, Covemng of the null ideal may have countable cojïnality, Fund. iklath. 166 109-136. (publication nmnber 592) iklR 2001m:03101 [$2] S. Shelah, Are and ] your cup of tea? Acta Math., to appear. (publication number 700) [vD] E. van Douwen, The itegers and topology, in: Handbook of Set-theoretic Topology (K. Kunen and J. Vaughan, eds.), North Holland, Amsterdam (1984), 111-167. iklR 87f:54008 IV] 3. E. Vaughan, Small uncountable cardinals and topology, in: Opea Problems in Topology (.J. van Mill and G. ikl. Rced, eds.), North-Holland (199(}), 195-218. THE (]RADUATE SCHOOL OF SCIENCE AND TECHNOLO(;Y. KOBE [INIVFISIT'. I{)KKO D3,1 1 1. NADA-KU. KOBE 657-8501, .I,PAN E-mail address: brendle@kurt, scitec, kobe-u, ac. jp TRANSACTIONS OF THE AMERICAN MATttEMATICAL SOCIETY Volume 355, Number 7, Pages 2651 2662 S 0002-9947103)03283-5 Article electronically published on March 14, 2003 CYCLICITY OF CM ELLIPTIC CURVES MODULO p ALINA CARMEN CO.IOCAIU ABSTRACT. Let E be an elliptic curve defined over Q and with complex multi- plication. For a prime p of good reduction, let. Ê be the reduction of E modulo p. We find the density of the primes p <_ oe for which E(Fp) is a cyclic group. An asymptotic formula for these primes had been obtained conditionally by J.-P. Serre in 1976, and unconditionally by Rare Murty in 1979. The aim of this paper is to give a new simpler unconditional proof of this asymptotic formula and also to provide explicit error terres in the formula. 1. INTRODUCTION Let E be an elliptic curve defined over Q and of conductor N. By a famous result of Mordell, the set E(Q) of Q-rational points of E is a finitely generated abelian group. The study of the free part of E(Q) is still one of the major problems in arithmetic geometry. Now, for a prime p of good reduction for E (that is, p N), we denote bv E the reduction of E modulo p. This is an clliptic curve defined over Fp, the finite field with p elements. Naturally, as in the rational case, one is interested in the study of the structure of the group E(p) of Fp-rational points of E. From classical theory, E(Fp) can be written as the product of two cyclic finite groups. Indeed, E(Fp) C_ E(Fp)[k] C_ Z/kZ Z/kZ, where Fp denotes the algebraic closure of Fp, k is a positive integer such that the order #E(Fp) of E(Fp) divides k, and E(Fp)[k] denotes the group of Fp-rational points of E annihilated by k. Early computations of Borosh, Moreno and Porta ([BMP]) showed that, in fact. for "many" primes p. the group E(]p) is cyclic. One expects this to be true for infinitely many primes p, as suggested by the elliptic curve analogue of Artin's primitive foot conjecture formulated by Lang and Trotter in 1977 (see [LT2]). Out goal in this paper is to provide an asymptotic formula, with explicit error terres, for the function f(x,Q) := #{p <_ x " p ,,E(Fp) cyclic}. in the case of an elliptic curve E defined over ( and with complex multiplication. In 1976 (see [Sel]), J. -P. Serre showed that C. Hooley's conditional method of proving Artin's conjecture on primitive roots (see [Ho, ch. 3]) can be adapted to estimate f(x, (). More precisely, let ( denote the algebraic closure of ( and Received by the editors July 24, 2002 and, in revised form, December 4, 2002. 2000 Mathematics Subject Classification. Primary 11G05; Secondary 11N36, 11G15, 11R45. Key words and phrases. Cyclicity of elliptic curves modulo p, complex multiplication, appli- cations of sieve methods. Research partially supported by an Ontario Graduate Scholarship. 9K1. (2003 American Mathematical Societv 2652 ALINA CARMEN CO IOCARU let Q(E[k]) denote the fiel(l obtain(,d by adjoining to Q the coordinates of the Q--rational points of E7 annihilated by k. T]lel, under the Generalized ienann Hypothesis (denotcd GRH) tbr the Dedekind zeta ftmctions of the division fields Q(E[k]) of E, Serre proves that, as ,r + OE, (1) f(x,Q) = fli,c + o . where li,r := f" 1 dt is the logarithmi« integral and k--I with p(-) denoting the M6bius fmlclion. X> recall hal for real-valued fimctions f and 9 # 0 we write f(z) = o(9(«)) to lllean that lim. I(*} _ O. Also. if 9 has p,,sitive values, we write g(x) -- ()r f « g fo mean that there exists a 1)ositive constant Ag(x) Vx. If lhe COllSlallt ,4 d('i)ends on S()llle qnantity B. then we may write /(x) = ()(9(x)) or f « g. In this paper, wllenever we write f(x) = O(g(x)) or .f « 9, we nlean thal thc imi)lied O-constants are absolute. If f « g « f, then we write f 9. In 1979 (sec [Mul. pp. 161-167]), Rare Murtv removed GRH in formula (1) tbr elliptic curves with complex multil)lication (deuoted CM). His proof uses clss fiekl theoretical properties of the division fields of CM elliptic curves, as well as a number fiekl version of the Bombieri-Vinogradov theorem (whose proof is based on the large sieve tbr number fields), lu 20(10 (sec [acC1]), the author proved formula (1) for elliptic curves without complex multiplication (denoted non-CM) under the assumption of a quasi-GRH (more precisely, a zero-free region of real part > 3/4 for the Dedekind zeta functions of Q(E[:])). For more history about /(x,Q) in i)oth the CM and non-CM cases we refer the reader to [acC1], [aeC2] and [Mu3]. In this paper we give a llPW simpler unconditional proof for the asymptotic formula for f(x, Q) in the complex multiplication case, and provide explicit error terlns in this fornmla. ç are proving the following: Theorem 1.1. Let E be a CM elliplic curve d«fin«d over Q, of condctor N ad with complex multiplicatio by the full tin9 of integers 0- of an imaginaw quadratic fi'eld Iç = Q ( ) . wh«re D is a positive square-free znt«ger. Then, as x (2) f(x,Q)=IElix+O.v (logx)(logloglogx) " or, more precisely, (3) f(x,Q)=IElix+O (log x)(log log og, 1Og log x . where the O-constant in (2) depends on N and the one in (3) is absolute. Corollary 1.2. Let E be a CM elliptic curve defined over Q, of conductor N and sch that Q(E[2]) ¢ Q. Then the smallest pme p { N for which Ê(Fp) is cyclic . ,« o (xp () ) . r v«e O-co. lit, was communicated to the author by Rare Murty that this result was obtained in 1979; however, it appeared in print onl3r in 1983. CYCLICITY OF CM ELLIPTIC CURVES MODULO p 2653 It is possible that the error t.erlllS iii Theorenl 1.1 Call be ilnproved, but this involves lnore sophisticated lnethods than the ones used iii our pal)er. \Ve relegate this te ftltllle research. 2. PRELIMINARIES 2.1. Notation. Given ai1 elliptic curve Æ defined over Q, p will always denote a prime of good reduction for ?. \Ve set ap := p + 1 - E(IFp) and say that p is of ordilmrv reduction if ap (, ami of supersilgular reduction if ap = 0. We denote bv zcp and the roots of t|le po|ynomial ,\' - aw\" + p Z[X]. If hot otherwise stated, q will denote a rational prime and k a positive integer: n(x) wi|l denote t|le nmnber of ratiollal prilnes G ,r; S will denote the cardinality of a set S: Ker ç wil| denote the kernel of a morphism ç. 2.2. Algebraie prelinfinaries, The followmg prelilninary |enmlas are well kmwn. but, for the sake of completeness, we inçlude thenl hel'e. Lemma 2.1. Let E be an elliptic curve defined over Q and of condu«'tor N. Let E[k] be the gro'u 1, of k-divi.sion points of E. Then (1) the ramified p,'in, es oJ'Q(E[k])/Q are divisors of hN; (2) assuming that E bas co'mplex multiplication and !« > "2. we hat,e O(k) " << [Q(E[k])-QI << k 2, where çb(k) dezotes the Euler flm«tiou. For proofs of this lelnlna the reader is referred te [Silvl p. 17"9] and [Sllv_. p. 1351. Lemma 2.2. Let E be an elliptic curve defined over Q and of conductor N. Usin9 the notation introduced in Section 2.1 we bave that, for a positive izde.qer !; and a prime p !; of 9ood redu«tion for E, 1» splits completely in Q(E[k])/Q if and only if zrp-- 1 is an algebraic integer. Pro@ \Ve recall that 7rp is the algebraic quadratic integer corresponding te the Frobenills endonlorphislll which we also denote by Since (p, kN) = 1, we have that 1' is ulralnified in Q(E[k])/Q (see part 1 of Lemna 2.1). By classical results in algebraic nulnber theory, p splits completely in Q(E[k])/Q if and only if 7rp]oe[] = 1, where 1 denotes t.he identity map. This last condition is equivalent te saying that Ker([h]) C Içer(wp- 1) as nmps Ê (-7) Ê (-7), where [k] is the multiplicatiol, by k map. Hence there exists an elliptic clIrve endomorphism b" E (IFp) Ê (-7) such that b o [k] = rrp - 1 (see [Silvl. Crolla.ry 4.11. p. 77]). This is equivalelt to saying that '- is ail a.lgebraic k integer. [] Lemma 2.3. Let E be a CM elliptic curve defined over an.d with complex mul- tiplication by an imagincwy quadratic field Iç. Then. for every prince p of ordinary good reduction for E, we have Q(Trp) = K. 2654 ALINA CARMEN COJOCARU Pro@ First we observe that Q(rp) c_ Endp(E) ®z Q c_ Endw(E ) ®z Q. Then we note that, since E has complex multiplication by/x, we have an embedding K C Endw(Ê ) ®z Q, and, moreover, since p is a prime of ordinary reduction, we actuallv have K = Endw(E ) ®z Q. Thus Q(rrp) c K for any prime p of ordinary reduction for E. But K is a degree 2 extension of Q, and so is Q(rp). This gives us the desired equality. [] Lenlma 2.3 describes a feature of Chi elliptic curves that will play a very impor- tant role in our unconditional estimates of f(x, Q). It actually describes one of the main differences between Chi and non-CM elliptic curves (see [LT1]). 2.3. Analytic prelilninaries. The next prelininary lemma is an application of the sieve of Eratosthenes, which we recall below. Theorem 2.4 (The sieve of Eratosthenes). Let .A be a set of natural nrnbers <_ x. and let 7 ) be a set of rational primes. To each pmme p distinguished residue classes modulo p. For any square-free integer d composed of primes of 7) we set .A(d) := {a .A a belogs to at least one of the w(p) residue classes modulo p for all and () := 1-[ (P)" p[d For a fixed real number z, we let S(.A. 7), z) be the number of elements a .A that do not belong to any of the distinguished residue classes modulo p for all p 7), p <_ z. and we set We assume that (1) there exists a real number X such that, for all square-free integers d com- posed of primes of (2) #A(d) = X- + Rd for some Rd = O(w(d)); w(p)logp _< clog z + O(1) for some positive constant c P pP p<_z Then S(.A,7),z) = XI"(z) + O (x(log z)C+l exp (--- where the implied O-constant is absolute. log For a proof of this result, see [Mu4, p. 141]. C CLICITY OF Chi ELLIPTIC CURVES MODULO p 2655 Lemma 2.5. Let x ]R and let D. k be fixed positive integers with k < v @ - 1. Then Sî := # p<_:r'p= k+l +D l'2forsomect,/3Z : O((-.X+l) v/71°gl°gx ) kv log --: --A " The implied O-constants are absolute. Proof. 1. Let us observe that the conditions p x and p = (ok + 1) 2 + D/32k 2 for some a, fl G Z imply Thus () s _< }2'# { where the sure '¢ -1 - x/Tl`. , -1 + v/7]l`: V V »v' 'v çZ, nz, #0. [ -1-v'@k '--l+v@]k rZ'(ah+l)2+D/3212apri'ne}" a E k ' k flZ , 7) := {pa rational prime" (p,k) = l, (pD) = l}, with (}) denoting the Legendre sylnbol nlodulo p. To each prinle p 7) we asso- ciate the residue classes (-1 + [3kD)k -1 (modp), where D is an integer such that D 2 = -D(modp) (let us observe that (ck + 1) 2 + D132q 2 = p imposes the conditions (=) = 1 and (p,k) = 1. and hence D and k-(modp) are well defined). For a fixed real number z we have oe . , V1Z'(at,'+l +D/32k 2aprime (5) < s(A,V,z) + ,(z) < S(A, 7), z) + z, with S(A, 7 ), z) defined as in the sieve of Eratosthenes. 2656 ALINA CARMEN COJOCARU Now we want to verifv that the hyl)otheses of Theorem 2.4 are sa.tisfied. Ele- mentarv estimates givc us #.A(d):=#{ce.A'(ok+l)2+D1321«2=O(nmde)}=2 +1 for ail square-free integers d composed of primes of P. Thus the first hypothesis of the sieve of Era.tosthenes is satisfied with w(d) = 9 and X = - + 1. Using glertens" theorem and recalling tha.t () = 1. hence that p splits completely in ,(v) ov _ og, _g: + o(1). vz pz Tlms hc sccoud hyl»ohcsis of lhe sicvc of Eroshcnes is stisfied wih c = 1. Thercfore, +l ll'(z)+O +l(logz) 2exp k log z " Il {z) = 1 - _< exp -2 « exp(-loglogz)- loz' by using the elemeutarv inequality 1 + t <_ exp(t) and. a.gain, Mertens theorem. Let us choose z such that Then O(X/-t- 1 k and so s(A. p. z) log z -- 3 log log .r" (log log x : (--. -t- 1) O \,og j -- (+,)o. V+I 1 ) k log.r(lo log x) 2 " ( ) + 0 k log x(log log x) 2 From (5) we obtain = (. + 1) 0 ( l°-g l°g'r log __2!_._ ) , which, used in (4), completes the proof of the first part of the lemma. 2. Sinfilar to the proof above CYCLICITY OF CM ELLIPTIC CURVES MODULO p 2657 We relnm'k t.hat for SI. and Sî. of the al)ove lelnlna we aetuallv have elelnentary log log estimates that are weaker than t.he ones given bv Lelnlna 2.5 onlv by a log x factor. The sieve has been invoked pre«isely fol" obtailfing this saving. Lemma 2.6. Keeping the notation of Lemma 2.5, we bave that, for any k and x, , () t. «< /,.-- +1 , where 1 <i<2. Pro@ We justify this estilnate for i = 1. The case i = 2 is l'esolved similarly. We observe hat the Colditions p <_ .r and p = (ak + 1) 2 + D/32k 2 for some a,/3 G Z give us - + 1 choices for o raid choices for/3. The lemlna follows. [] 3. THE PROOF OF THE THEOREM AND COROI,LARY As explained in [Mul. I,I»- 153-154], we have that E(IF») is cyclic if and only if p does not split COlnpletely in Q(E[q]) for any prime q ¢ p. Also, we have that if p _< .r and p splits eompletely in Q(E[k]) for SOlne /,', t|len k21(p + 1 - ap), and so, using Hasse's bolnld ap <_ 2V@, we obtain k _< 2v. Therefore, using the simple asylnptotic sieve, we tan write fez, Q) = N(x,g) + 0 (lll (x,y, 2v#-) ) , where ]V(z,y) := 4¢ {P _< x" pdoes llOt split COlnpletely iii ally Q(E[q])/Q,q <_ y}, 11I (x,v. 2v') := # {p < x" psplits conlpletely in someQ(E[q])/Q with .q < q < 2V}. and where y is a real nulnber to be chosen later, hl order to estinlate f(x, Q) we need to estimate ea«h of N(x, y) and ]Il(x, y. 2v ) and to «hoose the paralneter y appropriately. 3.1. Estimate for N(x, y). By the inchlsion-exchlsion principle we have ]V('F,V) = Et/-/(/')71-1( x" Q([/çl)/Q), where the sure is over ail square-free positive integers k < 2v whose prime divisors are < y, and where rl(z, Q( E[k]) /Q) := #{p < x "p splits COlllpletely in Q(E[k])/Q}. We estimate this sum by using the unconditional effective version of the Chebotarev density theorem as stated in [Mu2, p. 243] or [acC1. p. 337]. To do so, let us recall froln [Se2, p. 130] that if L/Q is a finite normal field extension that is ramified only at the primes p, p2,..., Pr.. then IL" Q-- l°g I disc(L/Q)l -< log[L" Q] + E logpj, 2658 ALINA CARMEN CO.IOCARU where [L'] and disc(L/) denote the degree and the discrilninal,t, respectively, of L/. V'e apply this result, together with Lemma 2.1, to the fields (E?[k]), whose degree and discrinfinant over we denote by ri(k) and d, respectively. D,'e get n(k)ldkl ' EE k8N 2 and ,,(k) 00g 141) 2 « k6 00g (»2v))2, and so the maximum of the two quantities above is « k8N 2. In order to apply the mmonditional effective Chebotarev density theoreln mentioned before we need to have kN « log x. Since k exp(2y), it is enough to choose (6) y = (log log x - 2 log N). Then, by the unconditional effective Chcbotarev density theorem, we obtain N(x, y) = n(k)] lix+ O a'exp -A tbr some effective positive constant A. To handle the error terre we use that n(k) « k 2 and that there m'e at lnost 2 .v square-free numbers composed of primes y. Then (r) N(x,y)= ç n(k)]lix+O N/4(logx) B for any positive constant B. 3.2. Estimate for M (x, y. 2). For rem numbers {, {2, we denote by ao («.{.{) the number of primes p x such that p bas ordilmry reduction and splits completelv in some Q(E[q]) with { q {2, and by M(x.{.{) the lmmber of primes p x such that p has supersingular reduction and splits completelv in some Q(E[q]) with { q {2- X write (s) M (, , ) = a o (x.v. ) + a * (,. v. e) and estimate each of the two terres. For the first one we observe that (9) where y<q_<2q rcï(x,Q(E[q])/Q) := #{p <_ x" ap J: 0 and p splits completely in Q(E[q])/Q}. By Lenmms 2.2 and 2.3 we obtain ,ï(x,Q(F[q])/Q) < # {p < Since the norm of rp in K/Q is p, we get { p -- 1 # px. q X ° -- rcp - 1 " (DI,- . q CYCLICITY OF CM ELLIPTIC CURVES MODULO p 2659 where Sq is Sq if-D = 2,3(mod4), and ,çq if-D = l(mod4), with Sq, Sq 2 as in Lelnma 2.5. Let us fix a real nulnber "u < vif- 1. Using the elementary estinmte for Sq given in Lelnnm 2.6, we obtain Z rï(.r, Q(E[q])/Q) <_ (10) << Sq (7 1) <q<2/ qv + 2x 1 v u<q<_2x/' x log log a" + v"u log u 1 On the other hand, using the estilnates for Sq given in Lemlna 2.5, we obtain y<q<_u Sq y<_q<_u y<qSu We choose u = log x *(loglogx 91ogN) (see (6)) and that D is bounded, since E and recall that y = g - _ bas Chi. Then, from (9), (10) and (11) we get (12) M°(x,g, 2vfT) =0 og (logx)(log N= (loglog v= ) For the second terln in (8) we have (3) where r(x,Q(E[q])/Q) := #{p _< x: ap = 0 and p splits completely in Q( E[q]) /Q }. We observe that if p is a prime of supersingular reduction that splits completely in some Q(E[q])/Q, then q = 2. Indeed. for such primes p and q we have, on the one hand, that q](p + 1 - ap) = (p + 1), and, on the other hand, that q](p - 1): thus q[2. Now we note that in the sure of (13) we run over q > y; thus, by out choice of y (see (6)), q 2. This implies that (14) M'(x,y,2v) =0. 2660 ALINA CARMEN COJOCARU #(k) | li X. --, t,(/) t,(k) V.,, .(k) » ,,(») - ,,(.) /--- ,,(»)' where k" which there exists a prime diviser q > y. Using part '2 of Lelmna 2.1 we get that lneans that the Slllll is over those positive square-free integers k for "p(k) x 1 lix «< log----TZZqt3/2 q>y t=l log :r log :r " -- ) 0og log -- ) .1" \ log log x ] log x J " log W- T}IllS (log x)y log B = O ((logx)(log = log x (15) f(:r,Q) Ielix +O (l°gz(l°gl°gw-) This completes t he proof of Theorem 1.1. 3.4. The proof of Corollary 1.2. First, let us recall that it was pointed out bv Serre (see [Mu3, p. 327]) that the density fE is positive if and only if Q(E[2]) ¢- Q. Now, we note that a necessary condition for forinula (15) to hold is that x _> e.p (V). Shen, if × e.p (V). the ,,in terre of (15) ,,-iii be bigger than the error terln. This proves the assertion of the corollary. 4. CONCLUDING REMARKS As mentioned in the proof of Corollary 1.2, the density [E is positive if and only if Q(E[2]) ¢- Q. For the sake of clarity, we explain this in what follows in the case of a CM elliptic curve/3 defined over Q. Naturally, in order te have Ioe ¢- 0 we need te assume Q(E[2]) ¢ Q, for otherwise the torsion part of E(Q) contains the Klein four group and se E(Fv) calmer be cvclic. The condition is also suflàcient. Te see this, let us first note that if Q(E[2]) ¢ Q, then [Q(E[2]) -Q] is 2, 3 or 6. We let K be the Ulfique abelian subextension contained in Q(E[2]). Also, we let Iç be the CM field of E. We recall that Iç(E[q]) = Q(E[q]) for any prime q > 3 (see [Mul, p. 165, Lelnlna 6]), and we observe that since K is a quadratic field and Iç2 is a cubic or a quadratic field, we have either K ç? K = Q or Iç = K. If K VI K = Q, Ch CLICITY OF CM ELLIPTIC CURVES MODULO p 2661 then using that K2 ç Q(E[2]) and Ix" ç Q(E[q]) for anv q _> 3. we deduce that the densitv of the primes p that do hOt split completely in anv of the fields Q(E[q]) is greater than or equal to the density of the prilnes p that do not split COlnpletely in K 2 and N. In other vords, ( )( )» c [ç.] 1 [¢.] _ . If N2 = N, then K ç Q(E[q]) for any prime q. and so the ,h'nsity of the primes p tha¢ do hot si,lit complotely in any of the fiehls Q(E[q]) is greater than or equal to the density of thc primes p that do hot si,lit COlnI)letely in N. In other xvords, ( I ) , f > 1 [Iç Q >- This completes the proof of the positivity of . The main significance of our uncomtitional proof of the asymptotic formula for f(x,Q) in the case of a ('gl elliptic curve lies in thc silnpli«ity of the tools that are used. Rare Mm'ty's initial proof avoided the (IH bv using a di«ult appli- cation of the large sieve for lmmber fields, nalnely a Bombieri-Vinogradov type result for mmfl)er fields. In our nev proof we use instead an application of the sieve of Eratosthenes, one of the silnt)lest sieves in nmnber theory. point out that this application of the sieve of Eratosthenes (Lemma 2.5) could be viewed as a Brun-Titchmarsh theorem for quadratic mmfl)er fields, since it gives nontrivial upper bomds for the lmllfler of (prilwipal) prilne ideals whose gelmrator satisfies congruente conditions. A result of this kind had been obtained in [Sch], but as an application of the large sieve for mmber fields, and «ould hm'e been used in our treatment of M(z. g. 2). Another significance of our new proof is that it provides explicit error terres, with absolute O-constants. As noted in Corollary 1.2. we can then deduce an unconditional upper bound for the smallest prime p for which E(Fp) is cyclic. Considerable improvements of this bound, mder GRH. will be discussed in an upconfing paper. Naturally, one can ask if our ideas can be explored finther and used in other related situations. For example, one could consider the question of determining the number of prime ideals for which the reduction of a CM elliptic curve defined over a number field gives a cvclic group. It seems that out tools can be used iii this situation. Another question is that of using the ideas of this paper in the case of a non-CM elliptic curve. At present no unconditiolml proof for the asymptotic formula for f(z, Q) is known in this situation, but, as mentioned in Section 1. onlv a proof based on a quasi-GRH assumption (see [acC1]). If we assume a variation of a conjecture of Lang and 'otter on the number of distinct fields Q(gp) obtained when p runs over primes of ordinm'v reduction for a non-CM elliptic curve (see [LT1]). then it turns out that we can follow the current CM approach even in 1 the non-C case. The dependence on the discriminant D of the estimates provided by Lemma 2.5 vill be more advantageous than the dependence on D provided bv Schaal's result memioned above. This is. again, ail asset of our new proof. Yet another related question is that of determilfing an asymptotic fornmla for the number of primes p for which the order of E(p) is square-free. The ideas of our paper can be successflflly used to answer this question if E is a Cg[ elliptic curve. The details of our last two claires will be given in different upcoming papers. 2662 ALINA CARNIEN COJOCARU ACKNOWLEDGEIvIENTS The results of thi paper are part of my doctoral thesis [acC2]. I express my deepest gratitude to my supervisor, Professor M. Rare Murty, for all his help and support. I ara also gratefld to Professor Erut Kani for usefld discussions on the algebraic prelimiuaries of the paper. tEFERENCES [acC1] A. C. Cojocaru, "On the cyclicity of the group of Fp-rational points of non-CM elliptic curves", Journal of Number Theory, vol. 96, no. 2, October 2002, pp. 335-350. [acÇ2] A. C Cojocaru. "Cyclicity of elliptic curves modulo p", Ph.D. thesis, Queen's University, Kingston, Cnrtda, 21|(12. [BlkIP] I. Borosh, C. J. Moreno, and H. Porta, "Elliptic curves over finite fields II", Mathematics of Computation. vol. 29, July 1975, pp. 951-964. lkIR 53:8067 [Ho] C. Hooley, "Applications of sieve methods to the theory of numbers". Cambridge University Press, 1976. IvIR 53:7976 [LT1] S. Lang and H. Trotter, "Frobenius distributions in GL2-extensions". Lecture Notes in Mathematics 504. Springer-Verlag, 1976. lkllq 58:27900 [LT2] S. Lang and H. Trotter, 'Primitive points on elliptic curves", Bulletin of the American Mathematical Society, vol. 83, uo. 2, March 1977, pp. 289-292. MR 55:308 [Mul] M. Rare Murty, "On Artin's conjecture", Journal of Number Theory, vol. 16, no. 2, April 1983, pp. 147-168. M1R 86f:11087 [Mu2] M. Raln Murty, "An analogue of Artin's conjecture for abelian extensions", Journal of Number Theory, vol. 18. no. 3, June 1984. pp. 241-248. lkIR 85j:11161 [lklu3] M. Rare Murty, "Artin's conjecture and elliptic analogues", Sieve Methods, Exponential Sums and their Applications in Number Theory (eds. G. R. H. Greaves. G. Harman, M. N. Huxley), Cambridge University Press, 1996, pp. 326-344. lkIR 2000a:11098 [Mu4] M. lRam Murty, "Problems in analytic number theory", Graduate Texts in Mathematics 206, Springer-Verlag, 2001. lkIF[ 2001k:11002 [Sch] W. Schaal, "On the large sieve method in algebraic number fields", Journal of Number Theory 2, 1970, pp. 249-270. MR 42:7626 [Sel] J. -P. Serre, "Résumé des cours de 1977-1978", Annuaire du Collège de France 1978, pp. 67-70. [Se2] J. -P. Serre, "Quelques applications du théorème de densité de Chebotarev", Inst. Hautes Etudes Sci. Publ. Math.. no. 54. 1981, pp. 123-201. M1R 83k:12011 [Silvl] J. H. Silverman, "The arithmetic of elliptic curves", Graduate Texts in Mathematics 106, Springer-Verlag, New York, 1986. lkIR 87g:11070 [Silv2] J. H. Silverman, "Advanced topics in the arithmetic of elliptic curves". Graduate Texts in Mathematics 151, Spriuger-Verlag, New York, 1994. lkIR 96b:11074 DEPARTMENT OF lklATHENIATICS AND STAT1STICS, QUEEN'S UNIVERSITY, K1NGSTON, ONTARIO, CNADA, KïL 3N6 E-mail address : alina@mast.queensu, ca Current address: The Fields Institute for Research iu lklathematical Sciences. 222 College Street, Toronto, Ontario, M5T 3J1, Canada E-mail address: alina@fields.utoronto, ca TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2663-2674 S 0002-9947(03)03194-5 Article electronically published on February 27, 2003 TAYLOR EXPANSION OF AN EISENSTEIN SERIES TONGHAI YANG ABSTRACT. In this paper, we give an explicit formula for the first two terres of the Taylor expansion of a classical Eisenstein series of weight 2k ÷ 1 for F0(q). Both the first ternl and the secoud terre have interesting arithmetic interpretations. \Ve apply the result to compute the central derivative of some Hecke L-functions. 0. INTRODUCTION Consider the classical Eisenstein series Iln(7"r) s, which bas a simple pole at s = 1. The wcll-known Kronecker limit formula gives a closed formula for the next terre (the constant terln) in terlns of the Dcdckind rkfunction and has a lot of applications in nulnber theorv. It seems natural and worthwhile to studv the saine question for more general Eisenstein series. For example, consider the Eiscnstein series (0.1) E(r,s) : e(d)(«r + d) -2k-' I,n(Tr) -k. -eP\Po(q) Here 7 = ( d b ), --q is a fundamental discriminant of an imaginary quadratic fiel& and e = (-q). This Eisenstein series was used in the celebrated work of Gross and Zagier ([GZ, Chapter IV]) to compute the central derivative of cuspidal modular forms of weight 2k + 2. The Eisenstein series is holomorphic (as a function of s) at the symlnetric center s = 0 with the leading terre (constant terre) given by a theta series via the Siegel-Weil formula. The almlogne of the Kronecker limit formula would be a closed formula for the central derivative at s = 0 -the main object of this paper. This would give a direct proof of [GZ, Proposition 4.5]. Another application is to give a closed formula for the central derivative of a family of Hccke L-series associated to Chi abelian varieties, which is very important in the arith- metic of Chi abelian varieties in view of the Birch and Swinnerton-Dver conjecture. This application will be given in section 4. We will also prove a transformation equation for the tangent line of the Eisenstein series at the center, which should be of independent interest. Received by the editors September 9, 2002. 2000 Mathematics Subject Classification. Primary llG05, llM20, 14H52. Key words and phrases. Kronecker formula, central derivative, elliptic curves, Eisenstein series. Partially supported by an ANIS Centennial fellowship and NSF grant DMS-0070476. @2003 American Mathematical Societ) 2664 TONGHAI YANG To make the exposition silnple, we assume that q > 3 is a prime congruent to 3 lnodulo 4. Let k = Q( Set A(s,) = -V( s+ 1)L(s, ) (0.2) and (0.3) /*(T,) = q A(s + 1, «)E(T, It is well known that E*(r, s) is hololnorphic. As in [GZ, Propositions 4.4 and 3.3], we define k (0.a) p(t) = () (-t),,, alld (0.5) qt.(t) __ e-t(u- 1)ku--du, t > 0. \;e rena.rk that p(-t) and qt.(t) are two «basic" solutions of the differential equa- tions (0.6) ,C"() + ( + t)C'() - »C() = 0. Filmlly. let p(n) be given by (0.7) <() = (,,)- Theorem 0.1. Let the notatwn be as above, and let h be the ideal class number of k. lI'rite r = u + iv. Then E*(r, 0)= v-(h + 2 __,p(n)p(4rcnv)e(nr)) and and ao(v) = h ( A'(1, e) 1) log(qv) + 2 A(1, e + Z a = (ordqt + 1)p(r01og q + (ordpr + 1)p(n/p)logp. (E)=--I The forlnulas should be colnpared to those for in [GZ, Propositions 4.4 and 4.5]. In fact, lnultiplying our forlnulas by the theta function in their paper and taking the trace would yield their formulas for ç. The method used here seelns to be more suitable fol" generalization. The proof is based on the observation that the Eisenstein series (0.1) can be split into two Eisenstein series. One of them is coherent, and it is easy to compute its value. It contributes little to the central TAYLOIR EXPANSION OF AN EISENSTEIN SERIES 2665 derivative. The other one is incohel'ent, contril)utes nothing to the value, a.nd its central derivative can be computed by the method of [KRY], where we dealt with the case k = 0. This consists of sections 1 and 2. In section 3, we study how the value and derivative behave mder the Fricke involution "r -1/qr and obtain the following fllnctional equation. One inter- esting point about the equatiou is that it basically follows from the definition of antomorphiç fornls (sec (3.2)). Theorem 0.2. The mo&dar forms E*(r, O) and E*'(r, O) satis.[y tbe.[ollowmg flmc- tioal eq-uatio: * 1 ( (-.o) E*'( -1 ol = i(v) 2"+ j=l --1 _1+1 log q ,c?*'(T, 0) " 3 Finally, let p be a canonical Hecke charact«r of weight l of k (see section 4 for the definition). It is associated to the C elliptic çurve 4(q) studicd bv Gl'OSS ([Gro]). When q = 3 mod 8, S. Miller and the author proved re«entlv lhat the central derivative L'(l. 13) 7 0 ([MY]). Since thc central derivative encodes vcry important information in the arithmctic of A(q), it is iml»ortalt to film a good formula for the central derivative. Standm'd calculation shows that the L-series L(s. I 3) is E(r, 2s) evaluated at a ÇM cycle. So Theorem Il.1 gives an explicit forlmlla for the central derivative L(1,13) (Crollary 4.2). 1. COHERENT AND INCOHERENT EISENSTEIN SEl:lIES Let G = SL2 over Q, and let B = TN be the standard Borel subgroup, where T is the standard maxinml split torus of B and N is the unipotent radical of B. Their ratiolml points are given bv and a- a } Consider the global induced representation I(s. e) = IndB(A) ] ] of ((N), where N is the ring of adèles of Q. Bv definition a section (s) I(s. «) satisfies (1.1) (n(b)m(a)g. s) = «(a)lal+(g, s) for a A* and b A. Let K = SL() and let K = SO(2)(). Associated to a standard section . which lneans that its restriction on IçIç is ildependent of s. one defines the Eisenstein series (1.2) E(g,s,) = (2g, s). eB(Q)ça(Q) It is absolutely convergent for e s > 1 and has a lneromorphic continuation to the whole complex s-plane. X consider three standard sections 0, i in this paper. For every priine p { q, let p I(s. ep) be the unique spherical section 2666 TONGHAI YANG such that (I)p(x) = 1 for every ,r C Kp = SL2(Zp). Let (I) C I(s, e) be the unique section of weight 2k + 1 in the sense that (1.3) (gko, s) = .(9, s) e(+)0 for everv ko ( cosO sinO = -iOcoO) K" Forp=q, let 4= cq be the Iwahori subgroup of Kq. Then % defines a character of .lq via (1.) «( cq As described in [KY, section 2], the subspace of I(s. Q) consisting of % eigenvec- i determined tors of Jq is two-dimensional and is spammd by the cell fimctions of q, by (1.5) qw/, s) = 5 0, where w0 = 1 and u, = w = . ç denote this subspace bv II (,lq, eq, s). A better basis for this subspace turns out to be given 1 (1.6) çq which are "eigenflmctions" of somc intcrtwining operator (see Lemma 2.2). Set (1.7) 0 o = q and = q p. Clearlv, 0 1 + -). For +iv with v . = ( + r=u >0.1et (1.8) g = ,,(),(). Then standard computation gives Proposition 1.1. Let te notatio be as above. Then .(, )= .-},(g, , 0) = -,,--i(*(g,,+) + *(g ,-)). 2 ' Here E*(g,s, ep) = q A(s + 1,e)E(g,s, dp) is the completion of the Eisenstein series E(g, s, (I)). As we will see in Proposition 2.4, the Eisensteiu series with (I) + behave almost as "even/odd" functions respectively, and both have nice functional equations. This is hot a coincidence. Indeed, ri'oto the point of view of representation theory, +(g, 0) is a coherent section in I(0, e) in the sense that it cornes from a global (two-dimensional) quadrat.ic space, while (I)-(g, 0) is an incoherent section in I(0, e), coming from a collection of inconsistent local quadrat.ic spaces. We refer to [Ku] for explanation of this terminology and for a general idea for computing the cen- tral derivative of incoherent Eisenstein series. Every section in I(0, e) is a linear combination of coherent and incoherent sections; we just made it explicit in this case. TAt LOR EXPANSION OF AN EISENSTEIN SERIES 2667 2. PROOF OF THEOREM 0.1 Let 0 = I-I 0p be the "canonicar' additive character of A via {e 2rix if p = oe, (,p(x) = e_2( ) if p -¢ x. Here A is the canonical map Qp Qp/Zp "- Q/Z. For a standard section = I-I (p E I(s, e) and d E Q, one defincs thc local Whittaker flmction IVa,p(9, s, ¢) = ¢(u,.,(b)9,s)Op(-db)ab. (2.1) Let (2.2) be its completion. \Ve also set lp(s) M M (s) is a nornlalized intertwning opcrat or from I(s. ) to I (-s, ). In gencral, an Eisenstein stries E*(9, s, ç) has a Fourier ext)ansion (2.3) E*(g,s, d with (2.4) E(9, s,4P) = q P for d ¢ 0 and -,-a «, (2.5) E(9, s,ç)=q A(s+l,¢)ç(g,s)+q 5I (s)ç(g,s). The local Whittaker integrals are computed in the next three lcmmas. Lemma 2.1 ([KRY, Lemma 2.4]). For a flnite prime number p ¢ q. II a,p1, s, p) = 0 vless ordp d > 0. h such a case, one bas ord. d v%,(1. . %) = (,(v)p-) r=O and M(s)((s) = Lp(s, ()(p(-s). Here çp is the unique spherical section defined in section 1. In particular where pp(d) -=- p(pord»d) for p < oe. Lemma 2.2. For p = q. one bas lV*,(Wl, s. '+ = one as (l+(q(d)q-S(°rdqd+l)) (±--- ) \ --q 0 if ordq d _> 0, if ordq d = - 1, otherwise 2668 TONGHAIYANG Mq(s)q 1 + = -'--------q . Pro@ The tir,st tbrnmla follows flore [KIY, (3.26)-(3.29)]. For the second formula, notice that AI,(s) is an itertwinig operator between eigenspaces ll'(Jq, eq, s) and :(./. ,-) of .,. . + b .I, (S)q = a q + for sonc constants a and b . Phgging in 9 = u'o and t,, and appl)ing the first formula, one gets the desired fbrnmla. Lema 2.3. Let = be the local section in I(s. e) defined b9(1.3). Il(g. l. t3, ) = / e-gx(._ 31- h) a-1 (.F - h)O-d.Æ -Fh>0 is Shi,tura «la fu,cliot for 9 > O. h N, atd e and e s'ucietttl9 large (2) For d > O. o« bas 1 ll.(g.0. ) = 2it,pk(4dt,)«(dr), where pk is deflned b (0.4). (3) For d < O , one has ll..(g.O, ) = O. and 1 1I 3. (g. o. ) = i,, q»(-4dt,)e( dT). where q is 9i,,e, by (0.5). j-sC (4) oE()() = i =0 +/2(-)e(-) Pro@ The proof is the sae as that of [KRY. Proposition 2.6] and is left to the reader. Proposition 2.4. (2.6) k k [[(j - )F*(g,-, e ) = : H(J + )F*(r,., Pro@ By Lenmlas 2.1-2.3. one has Now the proposition follows from the fimctional eqta.t.ions and E(9, s, ) = E(g,-s,l(s)). Here I(s) = M*(')A(s + 1. )- is the unnormalized intertwining operator from TAYLOR EXPANSION OF AN EISENSTEIN SERIES 2669 Theorem 2.5. One bas v (T,o., +) = 2(hq+ 2 ('.8) atd (2.9) 1 *l -- Proof. First. we observe that (2.10) H pp(d)(1 + %(d)) SillCe = p(Idl)(1 + %(d)) = 2p(d) 1 = H e(d)= sigl,(d)eq(d)II(-1) poe pi d Fornmla (2.8) is a special case of the Siegel-Weil forlnula. \Ve give a direct proof here using Lemmas 2.1-2.3. First, the lelnmas ilnply E(gT, 0. +) = 0 Ulfless d >_ 0 is an integer. \Vhen d > 0 is an integer, the lemlnas and (2.10) ilnply - l H Ea(g-, O, -) = q pp(d) 1 + eq(d)2iv½pk(4rdv)e(dr) 1 = 4vp(d)pk(47dv)e(dr). (2.11) Tlle saine |elnmas also ilnply E(g-, 0. +) = qA(1,¢)+(g,-.O) + q½M*(O)+(g-.O) 1 1 = hv + A(O, 1 = 2hv. This proves (2.8). As for (2.9), we again check terln by terre, and it is clear frolll tlle leIllllltlS that E'(g-, 0, -) = 0 uIfless d is an integer, which we assume froln now on. WheI d < O, II'd.,(9-,O,'b-) = 0 b3 Lemllla 2.3(3), and so (using Lemlnas 2.1-2.3 and (2.10)) "*' ' 0 -* , e'(9»0,-)=qu.tg, . oe)t,a(l 0, Il = -2,,½q(-v)«()/1 - )) Il 1 = -2v - p(-d)qk(-47rdv)e(dr), as desired. VVhen d > 0 and eqd) = 1, one has II'*.q(1.O, -) =0 and -1 IId,q(1,O, 4 -) -- v/(ordq d + 1) logq. 2670 TONGHAI YANG The saine computation using Letnnlas 2.1-2.3 and (2.10) yields *I -- 1 E e (g-, 0, (I)) = -2vpk(4nd'v)e(dr)(ordqd + 1)p(d)tog q 1 (2.12) = -2va,pk(4ndv)e(dr), since a = (ordq d+ 1)p(d)logq in this case. When d > 0 and eq(d) = -1. there is a prinle lld such that II)*t(1,0,t ) = pt(d) = 0 by 22.10). In this case, 1 (ordt d + 1) log l. The saine calulation yelds E d (g,O. -) = -2va,,p(4dv)«(d7), as desired. Finally, (2.13) with 22.14) go when d = 0, one has by the Saille lenlnlas, 1 t;(gT, s, +) = k . (G(s) + G(-0) k G(s) = (qv) A(1 + s. e) H( j + -). j=l (2.15) E('(g-, 0, (I)-)- "G--420) = hv½ Çlog(q.')+ 2-- This finishes the proof of (2.9). A(1,e) [] One has by Proposition 2.4. *(,o) lv--' * = E (g»O,,O +) 2 Proof of Theorem 0.1. (2.16) and (2.17) E*('r, 0) = v --½ lE *' l klE* ] L t..o.-)- ,=7 (-'°+) " Now Theorem 0.1 easily follows from Propositions 1.1 and 2.4 and Theorem 2.5. 3. PROOF OF THEOREM 0.2 By Proposition 1.1 and Fornmlas (2.16) and (2.17), Theorem 0.2 is equivalent to the identity ()2#+ ( -1 ï)(E'(9-,0.+,' (/*(g_,O,5+) =i l log q I k (gr.O,ê),]" (3.1) E'*' - t,s*'(-,o,<-)) To prove (3.1), one observes the following trivial but fmidamental identitv and computes both sides: (3.2) --1 * s; ( g«,,,I,-) = s; (w<,,-,,,I,+). TAYLOR EXPANSION OF AN EISENSTEIN SEIIES 2671 Here wf and tt, are the images of w = (ï 1 ) in C,(Af) and G(IR) respectivelv. The left-hand side of this identity is given by Lemma 3.1. For the right-hand side of (3.2), Olle has Lemlna 3.2. E*'(Wfgq-,O, dP-)] =i ½1ogq E*'(g-,O,(P-)]" Pro@ We verify these identities by comparing the Fourier coefficients E (WfgqT, s, dp +) with E(9, s, dp+). We nmy assulne that d is an integer bv Lemnms 2.1-2.3. Straightforward calculation using the saine lenmlas yields, for any integer d, (3.3) IV,»(Wygq¢,S, dp +) = F(d)II'.»(gT, s, dp +) with { ç if p { qoc, (3.4) F(d) = '- ifp= OE, _1_ 1 l+eq(d)q -st l+%(d)q_(+} if p = q. Here r = ordq d. We will verif.v the derivative part and leave the value part to the reader. First assume d -fl 0. It follows rioto (3.3) that E*" lE*'" ! ifeq(d)=-l, a_ (Wfgqr, 0, (I)-) = d tg-, 0, (P-) [,i if {[q (d) = 1. q ordq d+l When eq(d) = 1, one bas by (2.11) and (2.12), No d g-,O, dP-) _Ed(g,O,+)ordqd+ 1 logq. (3.5) = Ed (gr,O, P-) + logqEd(gr, O, +), Ee_ (WfgqT, O, (--) . ,, i q as desired. When eq(d) = -1 we have E(g-, ¢0, (I)+) = 0, and (3.5) still holds. It remains to check the constant term. Recall (2.13)-(2.15). Direct calculation using Lemmas 2.1-2.3 also gives (3.6) E)(Wfgqr,S, dP+)=:[: (q-G(s)+q-G(-s)). I]j=l (J -}- ) 2672 TONGHAI YANG Therefore, (3.7) E.,, , 2G'(0) .2G(0) l log q 0 uf9qr'O'dP-) =i k--, q-Z k! 2 i = iE'(g,-, 0. qb-) + logqE(gq-.O. b+), a.s expected, too. 4. L-SERIES Recall that q is a prime congruent to 3 modulo 4 and k = Q(v) is the associated imagi,mry quadratic field. Recall also ([Rob]) that a canoifical Hecke character of k of weight 2/," + 1 is a Hccke character p satisfying (1) The conductor of p is (2) p('2t) = It(ç() fiw an idem 1 relativcly l)rime to (3) t(aOk) = +o 2k+l. In this section, we will give an explicit forlmfla for thc central derivative of its L-fllnct.ion, which has dcep arithlnetic implications as mcntioned iii the introduc- tion. \Ve rcfer to [Gro] tbr the arithlnetics of elliptic curves associated to these Hccke characters (sec also [iXlY] and [Ya] and the refcrence there for more recent devclopments). For each ideal class C of k, we can define the partial L-series bv (4.) c(.,t,, c) = C, integral Of course, L(s, #) = -ce CL(k) L(s, t. C). The following proposition is standard. Proposition 4.1. Let Ç Che a primitive ideal of le relatively prime to 2q. and u,rite 9 = [a, b + V/ ] with a > O. b = O modq. 9 b+ v' 2aq L(s+l,'+ 1. p. C') -- (4.6) ¢(r) = ao(V) - 2 Z a,pk(4rrm')e(nr) - 2 Z p(-,,)q,.(-4rrm,)e(r-r). E* (r, 0) = v-O(r) t,("a) (.oE.o.l)2k+l (2v/)-L(2s + l,e)E(r,2). and Then Theorem 0.1 savs that and TA5 LOR EXPANSION OF AN EISENSTEIN SERIES 2673 Corollary 4.2. Let the notation be as in Proposition 4.1. (1) The central L-value is L'(k + 1, p., trivial) 7r 1 i Pro@ Only the second one needs a little explalmtion. When (-1)k() = -1 ,ve have L(k + 1. p. C) = {1 automaticallv and thus O.(r,a) = ). So Theorem 0.1 and Proposition 4.1 imply L'(k + 1, p, C) = When C is trivial, one tan take = Otc. In this case, a = 1 and and ttms çb(r,a)=çb(-+)._ b -- 1 lllod 1, 2q -- 2 In recent joint work with S. Miller ([MY]), xe proved that L'(1,p,trivial) > 0 when q = 3 mod 8 and k = 0. Combining that with Corollary 4.2, om has the following curious inequality: (4.7) 2674 TONGHAI YANG ACKNOWLEDGEMENT This work was inspired by joint work with Steve Kudla and Michael Rapoport. The author thanks theln for the inspiration. He thanks IRene Schoof for numerically verifving the formulae iii Corollary 4.2, which corrects a lnistake in an earlier version of this paper. Finally, |le thanks Dick Gross, Steve Miller. and David Rohrlich for stimulating discussions. EFERENCES [Gro] [KI:IY l [lohl [Shl [Y] B. Gross, Arithmetic on elliptic curves with complex multiplication, Lecture Notes in Math., no. 776, Springer-Verlag, Berlin, 1980. MR 81f:10041 B. Gross and D. Zagier, Heegner points and derivatives of L-series, Invent. Math. 84 (1986), 225-320. Mit 87j.-11057 S. Kudla, Central derivatives of Eisenstein semes and height pairings, Ann. Math. 146 (1997), 545-646. MR 99j: 11047 S. Kudla, M. Rapoport, and T.H. Yang, On the demvative of an Eisenstein series of weight one, Internat. Math. I/es. Nolices 7 (1999), 347-385. MR 2000b:11057 S. D. Millet and T. H. Yang, Non-vanishing of the central derivative of canonical Hecke L-functions, Math. Res. Letters 7 (2000), 263-277. MR 2001i:11058 D. IRohrlich, Roof numbers of Hecke L-functions of CM fields, Amer. J. Malh. 104 (1982), 517-543. MR 83j:12011 G. Shimura, Confluent hpergeometric functions on tube domains, Math. Ann. 260 (1982), 269-302. MR 84f:32040 T.H. Yang, On CM abelian varieties over imaginary quadratic fields, preprint. DEPARTMENT OF klATHEMATICS, UNIVERSITY OF VISCONSIN. kI.\DISON. VISCONSlN 53706 E-mail address: thyang@math.uisc, edu TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2675-2713 S 0002-9947(0303121-0 Article electronically published on Match 17, 2003 SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES ERIC FREEMAN ABSTRACT. ,Ve treat systems of real diagonal forms Fl(X), F(x),..., FR(x) of degree k, in s variables. Ve give a Iower bound so(R, k), which depends only on R and k, such that if s >_ s0(R. k) holds, then, under certain conditions on the forms, and for any positive rem number e, there is a nonzero integral sinul- taneous solution x E Z s of the system of Diophantine inequalities IFi(x)l < for 1 < i < R. In particular, our result is one of the first to treat systems of inequalities of even degree. The result is an extension of earlier work by the author on quadratic forms. Also, a restriction in that work is removed. which enables us to now treat combined systems of Diophantine equations and inequatities. 1. INTRODUCTION 1.1. Statement of main result. In 190. Schmidt [17] proved a far-reaching result about systems of Diophantine inequalities of odd degree. Given anv odd positive integers dl,...,d, Schmidt showed that there exists a positive integer si = si(d1 .... ,d), depending onlv on dl,...,d, with the following property: given any positive integer s _> s and any rem forms, or homogeneous polynomials, Gl(X) ..... G(x), in s variables, of respective degrees dl, .... d, and given any positive nmnber , there alwavs exists a nonzero integral vector y Z s satisfying the systenl (1.1) IGI(y)I < e, IG(y)I < e .... , IG(y)I < e. So, in other words, as long as the forms are ail of odd degree, and are defined in enough variables in terres only of the degrees, then there is a nonzero integral solution of the inequalities (1.1). Many particular classes of svstems of the type (1.1) bave been studied. For Diophantine inequalities of even degree, the situation is nmch different. There is no such general result as above for integral solutions of Diophantine in- equalities of even degree, and in fact there are few results at ail for inequalities of even degree. (However, results are known if one allows solutions in algebraic inte- gers in purely imaginary number fields. See Theorem 11.1 of [22].) In this article, we present one of the first results concerning systems of Diophantine inequalities of even degree, while at the saine rime removing a restriction from an earlier paper by the author, on quadratic Diophantine inequalities [10]. We are now able to remove Received by the editors October 15, 2001. 2000 Mathematics Subject Classification. Primary 11D75; Secondary llD41, 11D72, 11P55. Key words and phrases. Combined systems of Diophantine equations and inequalities, forms in many variables, applications of the Hardy-Littlewood method. The author was supported by an NSF Postdoctoral Fellowship. (2003 American MathematicM Societv '2-676 ERIC FREEMAN mo (/, &, ) = / and (1.5) the l'estrict.ion bv com|)ilfilig the powerful ideas of B(mtkus and G6tze [3] with the techniques of Nadesalingaln and Pitlnan [16], and bv adapting our previous work in [10] and [11] to treat the miuor arcs properly. To state out first result, we require SOlne notation and definitions. X shall be working with systems of diagonal forlns Fi(x) given by (1.2) Fi(x) For systems of forms Fi as in (1.2), we define the coefficient matrix of the system F to be the matrix (1.3) A = (AO)<i<n. ljs For 1 j s, we denotc the jth çohnnn of A by Aj. Now suppose that ,1 is a subset of the set of indices {1,2 .... to I)c the submatrix of A consisting of the cohmms Ni with j J, and we define r(A.) to 1)e the tank of the matrix Aj. Finally, if x N satisfies Fi(x) = 0 for 1 < i < R and the matrix lSjSs is of full tank, then we sav that x is a nonsingular solution of the system F. Now, for integers R and k and any rem nulnber u. we define the flmctions (.4) rein (4R + 4R + 1,384 log 16R + 5) if k = 2 Rk log 2k lg2 + uklog(Rlog2k) if k is odd and k 3 h [48k log 3Rk 2] if k 3. { ( 5 ) no(k,u) = nfin 2k-t - 1, k(logh+loglogk+2)+ ukloglogk log k V'e can now state out first result. if k = 2 if k>3. Theoreln 1.1. Suppose that k is an inte9er with k >_ 2 and that R is a positive integer. There are absolute real positive constants C and C2 for which the following property holds: Suppose that g is an integer satisfying (1.6) « lllgX(?/,0 (/?,',dl) ,N0 (]x'.d2)) . Suppose that s is an integer satisfyin9 s >_ gR. For 1 < i < R. suppose that Fi(x) is a real diagonal form of de9ree k, as in (1.2). Let A be the coefficient matrix of the system F, as in (1.3). Assume that the followin 9 two conditions are satisfied: (i) Either k is odd, or there exists a real nonsin9ular solution y of the system FI(y) = F2(y) ----- .... F/¢(y) ----- 0. (ii) For every subset J C_ {1,2 ..... s}. one hs lJl <_s-e(R-r(Aj)). Fix any positive real number e. Then there is a nonzero inte9ral solution x G Z of the system (1.7) IF(x)l < for 1 < i < R. SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2677 For general even/% Theoreln 1.1 is one of the first results of its kind to our knowl- edge. We note that at least somc conditions similar to Ci) and (ii) are necessary, as may be seen by considering the examples given after Theorem 2 of [7]. For odd/,', we note that Theorem 1.1 is hot very much of an improvement be.vond that given bv Nadesalingam and Pitman [16], and could presumably be obtailmd by combining their methods with results of Vaughan [19], [20] a.nd work of Woolev [23]. We also observe that, following the method of Section 7.2 of [16], we could relnove condition (ii) for odd/," if we chose to do so. As well, our lnethod of proof shows that mlder the conditions of Theorem 1.1, ve can give a lower 1)ound of the expected order of magnitude, P-, for the nmnber of solutions of (1.7) in a box of size P. ff»r all sufficiently large P. This was hot previously knowm even in the special case of systelns of inequalities of odd degree. We emphasize that when using our methods, condition (ii) is necessary to obtain this lower bound. Presulnably, one could also give an as3mptotic formula for the nulnber of solutions, bv combining with the methods of [12]. Note that we bave ex«luded the case/," = 1 from the statenlent of the theorem. In this case, oto" knowledgc is ranch l»etter. For h = 1. if one has 8 >_ /ï' -1- 1, a. nonzero solution of (1.7) may bc follnd using a l»ox principle, whcther or hot condition (ii) holds. (Sec the Lemma in [4].) For k = 1. one can also find many solutions of (1.7) in a box of size P. (Sec Lemma 1 of [9].) 1.2. Combined systems of Diophantine equatiolls and inequalities. \Ve note now that it would actuallv bc faMv routine to give at lcast ont result on inequalities of even degrec; one could simply generalize the work in [10] by con> bining those techniques with the methods in [11]. However. such a generalization would exclude many important classes of svstems of inequalities, fol" example those in which some of the forms are integral. We were forced to exclude such systems iii [10] because of the methods we used. In out current work, we are able to treat these formerly excluded systems. We give some background to more full,v expla.in. In [10], we considered simultaneous systems of diagonal quadratic Diophantine inequalities. Fol" a positive integer R. definc, for 1 _< i _< R, the real quadratic forms Q(x) = a,î + a2«g +... + a,«. It was proved in [10] that for every positive real lmmber e, under certain conditions on the system of forms Q, Q_ ..... Qn, there is an integral vector x Z \ {0} such that one has In that paper, one of the conditions we assumed was the following. (Sec condition (iii) of Theorem 3 of [10].) (1.8) For each choice of (fit,/32 ..... 3n) C liRn \ {0}, there is at least one coefficient of fltQt + f12Q2 + ... +/3nQn that is irrational. This condition allowed us to use a modification of the remarkable work of Bentkus and G6tze [3], but excludes certain important systems ff'oto consideration. The restriction (1.8) rules out systelns in which one or lnore of the forms is an integral form, and also any system iii which any nontrivial linear combination of the forlns 2678 ERIC FREEMAN is a.n integral form. The centra.l new contribution of this paper is in removing the condition (1.8). We now state a lnore technical and lnore genera.l version of Theorem 1.1. We require lnore nota.tion. For a. rem vector = (/31,...,/3n) E N u and a system G of forms G1 (x), G2(x) ..... Gn(x), we define the forin (t-G)(x) =/C (x) + 2C(x) + ... + Also, for rem nulnbers z, wc define e() = e2. Theorem 1.2. Suppose that k is an integer with k >_ 2 and that r and R are integers with R >_ 1 and 0 <_ r <_ R. Then there are absolute positive real constants C and C with the following propertg: Deflne no (k, u) as in (1.5). Suppose that f is an integer satisfying (1.9) « >_ ,,0 (k,02) . Let s be an integer witb s >_ fR. Ah'o suppose, for 1 < i < H. tbat (x) = a,l.-î + a: +... + is a diagonal form witb real coefficients. Let ,4 be the coefficient matmx of the system F, as in (1.3). Assume that the following four conditions are satzsfled: (i) Either k is odd. or tlere exists a real nonsingular solution y of the system Fl(y) = F(y) ..... Fn(y) = 0. (ii) For every subset J ç {1.2 .... ,s}. ov bas [J[ _< s-f(R- r(Ar)). (iii) Tbe forms F, F ..... Ff bave integer coefficients: also. if = (a. aee .... , oR) IR n and o F is a rational form, tben ar+ = ct+2 ..... an = 0. (iv) If r >_ 1 bolds, tben there is a positive real constant c(F) suct that one has q=l a:(a ..... ar,q)=l 3=1 oe:l i=1 l <_ai <_q (l_<i_<r) Fix any positive real number e. Then there s a nonzero integral solution x Z of the system Fi(y) = 0 for l<i<r, (1.10) IFi(y)l < e for r+l < i < R. Moreover, if we define m0 (r. k, u) as in (1.4) avd we assume that tbe condition e II 0 (r, k, d 1) holds, then we may omit condition from OUF assumptions. Some discussion o( the condition (iii) is warranted here, since it is the most important distinction between Theorems 1.1 and 1.2. In Theorem 1.1, we consider systems of inequalities (1.7). Now, if one chooses e < 1. and F(x). sa.',', is actually an integral form, then the system (1.7) reduces to the system t:(y) = 0. IFi(y)l<e for 2<i<R. SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2679 So, in this case, the systeln of R inequalities aetually reduces to a systeln of one equation and R- 1 inequalities. A similar reduction occurs if some nontrivial real linear Colnbilation of the forlns ri, soEv ctF1 + c'.)F2 + ... + aRFR, is an integral fornl. In these situations, one might say that there is açtually ail equation hidden iii the system of inequalities. The conditi(m (iii) ensures that there are actually r equations in the systeln and R - r "truc" inequalities. It is helpflll to ensure that there are hOt any lnore "hidden" equations becmlse it turns out that one requires more variables to treat the system if there are more equations present. One lna.v think of Theorem 1.2, more or less, as the sub-case of Theorem 1.1 in which there are exactly r equations present in a systeln of /? inequalities. \Ve note that the second clause of condition (iii) is vacuous if r = R holds. One might question if, in condition (iii), the terln -F could be replaced by the terre aï+Fï+ + c+2F+2 + ... + cRFR, to give a slightly weaker condition asserting that r equations are "'hiddelf" in the systeln. It turns out that one tan not, as ma.v be seen by considering the example F(x) = F(x) = k "]['8 ' where a 3, Ci4,..., as are anv integers. Here, for r = 1 a.nd R = 2, COl(lition (iii) does hOt hold, since (1/x/) (F2 - F1) is an integral forln, filial thlls tac systenl (1.10) is equivalent iii this case, for Slnall e, to the s.vsteln 1 Fl(x) = (F:(x) - Fl(X)) = 0; on the other hand, for any nonzero real nulnber a2, the forln ct2F 2 is not a rational forlnsincetheratio ((3+v)/(2+v/-))isirrational:sothissystelndoesnot satisfy the suggested replacement condition. So the putative replacelnent condition is hOt strong enough. We now discuss condition (iv). The tenn is the so-called singular sertes, and condition (iv) silnply states that it is bounded below by a positive constant, a necessary precondition when using the Hardy-Littlewood method. As the last sentence of Theoreln 1.2 states, we could have omitted the condition from our assumptions iii favor of a lower bound for /. However, since the consideration of the singular sertes is not our central lotus in this work, we have chosen to include condition (iv) so that out result can be improved immediately and transparently if improvements arise concerning the singular sertes and the p-adic problem. This should certainly be possible in the case k = 2, for example. Also, we wish to clearly indicate that tUe condition /? _> m0 (r,k,C) is needed Olfly because of tUe p-adic problem. 1.3. Related results. çVe now compare out work with other results. For even k, Theorem 1.1 is an analogue of a result of Davenport and Lewis, coucerning systelns of Diophantine equations of even degree. (See Theoreln 2 of [7].) They assmne that 2680 ERIC FREEMAN a svstem of diagonal equations FI(x) = F2(x) ..... FR(x) = 0. of even degree k. with k > 2, has a real nonsingular solution and also that foi" 1 _< S _< R, every set of S independent integral linear combinations of Fa ..... FR contains at least (1.11) [4RSk 2 log (3Rk2)] variables that aI)p«'ar explicitly. Under these conditions, the system of equations bas a nonzero integral solution. \Ve note that if one replaces the quantity in (1.11) bv Sé, and restricts to integral forms, then one can show that their second condition is equivalent to condition (ii) of Theorem 1,1. Nadesalingam and Pitman [16] proved that any R real diagonal Diophantine inequalities of odd degree k, with k _> 13, iii .s variables with . > 3Rk 2 log (3Rk) bave a nonzero solution. We note that thev do hot require anv condition that is sinfilar to condition (il) of Theowm 1.1. Also, we observe that they could certainly have used their mothods to obtain similar results, although with a different lower bomd foi" s, in the cases k < 13, but iii order to streamline the presentation thev did hot do so. Finally, we note that Briideln and Cook [5] bave given a lesult on systems of diagonal Diophantine inequalities of odd degree. Under certain conditions on the coefficient matrix of the system, they show that there is a nonzero solution of the system of inequalities. They require an assmnption similar to condition (ii) of Theorem 1.1 and also a condition that is stronger than (1.8). The lmmber of variables they require is on the order of Rn0 (k. C2). We also note that they call find a lower bomd of the expected order of inagnitude foi" the mnnber of solutions of their system iii a box of size t » for a sequence of positive P tending to infinity. although hot for all large P. as out treatment provides. 1.4. Methods used. The general strategy of the proof is to combine the method of I3entkus and G6tze [3], which is very effective for Diophantine inequalities, with the techniques that Nadesalingam and Pitman [16] use to treat combined svs- teins of Diophantine equations and inequalities. We remark that the techniques of Nadesalingam and Pitman are themselves a combination of the Hardy-Littlewood method and the Davenport-Heilbronn method. Using the techniques of Nade- salingam and Pitmm allows us to treat those systems of inequalities that contain 'qliddeff" equations. Fol" those who are fmniliar with their argmnent, we note that we do hot have a so-called residual set in our proof, as in their paper. One other crucial result is needed, and this involves showing that, on the minor arcs, oui" exponential sulns are smaller than the trivial bound. Iii previous work on these t,vpes of problems, including [10], [111, and essentially also [3], such a result was achieved by splitting the minor arcs into two regions and handling each separately. In this paper, we handle both of these regions together, which is hot onlv cleaner, but also seems to be necessary here. I would like to thank Scott Parsell foi" showing me how to improve Lemma 6.1. I would also like to thank Michael Knapp and Professor Woolev for indicating to me how to prove part of Lemma 8.5. SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2681 '2. DEDIÇTION OF THEOIEM 1.1 FFIOlkl THEOIEM 1.2 The lmlk of this pal»er is dedicated te proving Theoreln 1.2. In this section. however, we delnonstrate how Theoreln 1.2 implies Theoreln 1.1. Te this end, we consider a system le of real diagonal ferres F1, Fa,..., FR as in Theorenl 1.1. We give a definition first. Suppose tlTat G1, G9 ..... GR and lit, ici.,_ ..... li/ are two svstenls of forlns. If there exists a set of R lilmarly independent real vectors /3, fie ..... /3R E IR R such that H(x)=/3-G fir 1 <i< R, then we say that the svstem H is equivalent te the system G, whi«h we denote by G H. It is easv te check thaI this is in fa«t an equivalence relation. \Ve olserve as well that if G is, in particular, a system of diagonal fol'mS, and G , H holds. then H is also a system of diagonal ferres. For anv system of ferres G. we define z(G) te be the lmml)er of ferres among GI,G_ ..... Gn that are integral, that is, whose coeffi«ients are all integers. New for out systent of ferres F, we define r = r(F) = ma.x In othcr words, r(F) is shllply the lllaXillllllll llllllll)er of forlllS that are integral in allV systenl G equivalent te F. \Ve clearlv have 0 _< r _< B. New suppose that G is a system equivalent te F and that G has r integral ferres. Se there exist H real linearly indcpendent vectors/3./3e,...,/3/ E IR R such that Gi =/3, - F for 1 _< i _< R, and also the systeln G contains r integral ferres. Bv relabeling if necessary, we mav aSSllllle that G1, G2 ..... Gr are integral forlllS. V,e new show that conditions (i)-{iv) of Theorenl 1.2 hold for this systenl Cf. Then we will apply Theorenl 1.'2 te Cf. and we will see that the nonzero sohttion of the systenl G is also, under certain Coliditions, a solution of the svstelll F. Silice F is equivalent te Cf. if thc coefficient lnatrix of F is .4. then the coeflï«ient matrix of Cl is TA for the nonsingular R × R ulatrix T with rows/3,. Thus. for any subset J C_ { 1, 2 .... , s}, we have (2.1) r((TA)s) = r (TA.,) = r(As). The systenl F has a nonsingular sohltion {1.2 .... ,s} v«ith IJI = H satisfying r(Aa) x if and onlv if there is a sul)set ,1 of R and H md -j: 0. Thus the existence jd of a real nOlsingtllar solution for G follows ff'oto the existence of such a sohltion for F. Se condition (i) holds for G. By (2.1), it is easy te see that condition (ii) holds for the coeflïicient lnatrix of G. becallse it holds fol the coeflïicient lnatrix of F. New we turn te showing that condition (iii) of Theorem 1.2 holds for the system ' .... a) IR R is a real ve«tor such that G. Te this end, suppose that et' = (a, au ! i et'- G is a rational fonn. We need te show that o,.+1 = Or+ 2 .... = O R = 0 holds. This holds vacuouslv if r = R. For r < R. clearing denonlinators, we see that there is a nonzero integer n such that. defining et = et', we have et-G = .et'-G Z[x]. Since is llOllzero, to prove that a+ 1¢ = a+ 2¢ .... - = 0 Re = 0 holds, it is enough to prove that we have c+l = Ctr+2 ..... OR = 0. So SUl)pose that this is hot the case. Lett.ing el, e ..... eR 1)e the standard relit basis for IR R, we then have that e.e2 ..... er. et are r+ 1 lilmarly ilMependent 2682 ERIC FREEMAN vectors in R. çVe inay thus extend this set to a basis, sa.,," 9',9', .... 3' of N n, with i = ci for 1 i r, and r+l = " Then the system of forms 1 " G, 2 " G .... ,n - G is equiwdent to G, and thus in turn to F. But its first r + 1 forms are integral. This contradicts the definition of r(F), and thus we lnust in fact have -r+l r+2 "'" R 0. Thus condition (iii) holds for G. since satisfies (1.6), wP havÇ g oE m0 (,k, Çl). alld tlms we certainlv Now, have g mo (r, k, Ç). whence by the final sentence of Theorem 1.2, condition (iv) is UlmeCeSSaly. Since we have also secn that conditions (i)-(iii) of Theorem 1.2 hold. we lnay apply Theoreln 1.2 to the system G. Before doing so, we give some more notation. For a vector x = (z, w ..... x) N , define Ixl = ,l,aX Ixl. l<j<s For an R x s matrix BI = (tlij)l<i<R, We define l<j<s We note that the notation differs slightly from that used b5 some other authors, for exalnple Nadesalingam and Pitlnan [16]. Similarly, for a system F as in (1.2), we define IIFII = II/'XR Iijl Defining the matrix T as above, we certainly have det(T) ¢ 0 and I[Tll ¢ 0: so we may apply Theorem 1.2 to the system G with e replaced by the quantity (Idet(T)l¢) / (RIITII-). X obtain a nonzero integral solution x e Z s of the system Idet(T)l la(x)l < (1 < i < ). IITII -1 By Çramer's rule and Hadalnard's rule, iI follows that x is also a solution of the systeln (1.7), whence Theoreln 1.1 follows. çç now turn to the proof of Theoreln 1.2, which colnprises the rest of the paper. 3. INITIAL ]EDUCTIONS In this section, we reduce the problem of proving Theorem 1.2 to the consider- ation of a systeln of forms as in Theorem 1.2. but under a few more restrictions. This will lnake our application of the Hardy-Littlewood method casier. We first note that by considering the forms e-lFi, it is enough to consider only the case çVe can also assume that we have IIFII > 1. If this were not the case, then (1.0.0 ..... O) would be a solution of the system (1.10) and we would be done. SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2683 We now quote a lenmia, which seems to have been first used in this field bv Low, Pitmau and Wolff. (Sec Lemma 1 of [13].) It is actually a special case of a result on matroids, apparently due originally to Edmonds [8]. A proof tan also be fotmd in Aigner. (Sec Proposition 6.45 of [1].) Lelnlna 3.1. Let A be an Rxs "matrix over a field K and let w be a positive iuteger. The matrix A bas a R x Rt, pa, rtitiomble s'ubmatriz (that is. A icludes w disjoint Rx R submatrzces that are to,sitgula, r over K ) if a,td o'nlg if the followin 9 co,tditiot is satisfied: (3.1) [J[ s - u,(R- r(Aj)) for all sub.s'ets J ç {1.'2 ...... }. To be clear, by including w disjoint R × R nonsingular suhmatrices, we mean that there is solne pernmtation of the cohmms so that the first R cohmms form a nonsingular matrix, as do the second R colunms, and so on. Note that the condition (3.1) is exactlv condition (il) of Theorem 1.2 in thc case « = (. Thus we may apply the lcmma to Ihe coefficient matrix A of the svstem F, with the choice u' = (. Thercfore, A has ill R X ( partitionable sulmiatrix. By relabeling vmial)les if ne«ssary, we lnay write (3.2) A=[A A .... Ce Acn+ A«n+2 ... A, ], where A. is an R x R submatrix for 1 N v N g and where (a.a) A.=dt(A,,)l¢0 for lvSC. Now considcr the system F in the case that k is odd. show that F has a real nonsingular sohltion. Since A has the form (3.2), and (3.3) holds, one can sec that F is equivalent to a system G with coefficient matrix B such that the left-hand R x 2R sulmmtrix of B has the form here I is the R x R identity matrix, and B.2 is a nonsingular R x R matrix. can find real nunlbers z+a, z+2,..., z2 satis-ing zR+ [ - 1 ZR+ 2 -1 2 = . z2 -1 Nowlet zj = 1 for 1 j R, and let zj = 0 forj > 2R. Then for 1 j s, define z = zj , which is always real, since k is odd. Setting x = (Zl,Z2 ..... one can observe that Gi(x) = 0 for 1 < i < R. Now the left-hand R x R matrix of -(OG has detenninant " -1 -1 -1 which is nonzero. Thus the h" " " " Y system G has a real nonsingular solution, whence, as in Section 2. the svstem F does as well. Thus, whether k is odd or even, we know that there is a nonsingular solution of the system F(x)=0 (1 <i<). now show that there is a real nonsingular solution whose conlponents are all positive. As noted above, there is a subset J = {j,j2 ..... j} Ç {1, 2 ..... s} with 2684 EIRIC FIREEMAN ]J] = R and a real vector x E ]s suçh that we have det(Aj) ¢ 0 and H xj ¢ O. Noxv, tbr 1 < i < R, we define the linear fornl (3.4) Li(y) = Li(y,Y2 .... ,Ys) = AijYj. j:l » for 1 < j < s, we see that th('re is a rem vector z = (z ..... z) O11 setting zj = ai _ _ R such that zj ¢ 11 holds fl»rj .1, and we have (3.5) L.i(z)=0 for 1 <i<R. Nov, if k is even. oto choice of z ensures that we have zj 0 for 1 j s. If k is odd, thon for each j, we lllay if necessary replace zj by -zj, and replace the ('oeEicients ij by --/kij for 1 i , and consider the resulting system, hl this lllilllllel', Wç IIIHV ellSllre that we bave a solution z of (3.5) with zj (I for 1 j s and zj > (I for j E ,I. Note that Colditions (il) and (iii) of Theoreln 1.2 and «onditims (3.2) and (3.3) are mmffe«tcd. Con(litiol (iv) is also unaffected, since q (" tl'l'k zjfl.) the Slllll ,Ç is always rea.1 for odd k with A O E, which mav be J'=[ fil scen lv subsfimhg -« for m h tire SUln. Now suppose tha z = 0 for some j0 safisfying 1 j0 s. %%% clearlv bave fix a positive rem mmlwr ? wih nllll Shce Ad is nonsingular, thee is rem vtor w = (w,..., teR) such that we lmve A.w = -Q Aço. Bv Crmner's rule md Hadmnmd's rule, we certinlv hve I'1 -< Id«,(A,)l <- m. 1 o t < < . Now defilm , zj + tt'i forj =ji E J for j = jo fol" j ¢ J LA {jo }. Writing z' ( z'. "' = -2 ..... Z), we have Li(z t) = Li(z) =0 fol" 1 <i < . Also, we hae ' > 0 for j JU{j0}. All ofthe other COllpOlWnts ofe' are equal to the respective COlnpOlmnts of e: so we bave replaced our rem nonsingular solution by a rem nonsingular solution that bas Olm more positive COlnpolmnt and tlmt still satisfios tlm çondition zj > 0 for j 3. poating this proçss as manv as (s - R) rimes, we can find a lonsingular real solution e = (z ..... z) with z > 0 for ljs. Thus, scaling if necessal% ve lnaV choose a l'eal lmlnber and rem munbers z, ï2 ..... c, tbat satisfv t for 1 <j<s, and 0<Nz5 _ _ (.) L(e)=0 for l<i<R. SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2685 To sure up, iii this section, we have delnonstrated that to prove Theorem 1.2, it is enough to consider a system of forms F1, F2,..., F/ as iii Theorem 1.2, with the added assumptions that IIFII > 1 hohts, that thc coefficient matrix .4 of the system satisfies the conditions (3.2) and (3.3), and that there is a real ve('tor z satisf,ving (3.6). Iii seçtions 4 9, we l)rove Theoreni 1.2 mlder these ad(titional assmnl)tions. 4. THE DAVENPOIRT-HEILBRONN IIETtlOD" TttE ,¢";ETI:P Now we proceed with the proof of Theorenl 1.2 under the additional assmnptions we ruade a.bove. ' shall essentially use the Hardy-Littlewood method, in an involved form. X> combilie the methods of Bentkus and G6tze [3] with those of Nadesalingam and Pitlnan [16]. note that thl'oughout the pal»er, implicil constants in the notation o() and O() and << a.nd >> lllay dot)end on R. s, k, , t, the coecients of the tllllS F1 ..... F, and the real ve«tor z. > cousider the lmmber of solutions of the svstem (4.i) F,.(y) = 0 f,,r 1 < i < ,', IF(y)l In the usual fashion, we use a real-valued, even kerncl flm('tion K N + N to give a lower bomM for the nmnber of integral solutions of the svsWm {4.1) in a certain range. Define such a flm«tion K, fi)r any real mlmber a,, bv ., By Lemma 14.1 of [2], fi)r a.ny real nmnber u, the function K satisfies the identity ] { ,, ,f ,,, (4.3) ¢(u) = «(,,)K(/)S = I -I"1 if I1 < 1. The function K satisfies, for real nmnbel'S , the bomd (4.4) % will a.lso use the identitv t {1 i,,: (4.5) (')'« = 0 if ,, e z {0}. Now for positive real numbers P and Q satisfying Q P, we define the so-called Q-smooth lmmbers tobe the set A(P, Q) = {x Fix a positive real number q, to be chosen later, so that it will satisfy the require- ments of Lemmas 5.5 and 6.1. Then for real numbers a and P with P > 1, we define the exponential sure g{a) over the smooth immbers bv (4.6) g(a)=g(a,P)= e (a.r). x(P, Pv Also, for 1 j s, and real veçtors a G , we define the lillear forlns (4.7) i=1 2686 ERIC FREEMAN Then we also define, for o G I t and rem nmnbers P with P _> 1, aud for 1 _< j _< s, the flnctious gj(o 0 = gj(o, P) = g (Aj(o), P) . (o.s) We define as well n" = [0, 11 x R-. Now let .lU(P) be the number of solutions of the systenl (4.1) with xj E (P, P) for 1 j s. By using the property (4.3) of the fimction K(a) and the identitv (4.5), one can sec that we have xj A(P,P n) i:l i=r+l ljs observe that this last renmrk is justified by the fact that the integral converges absolutcly, which follows froll (4.4), whence we mav write the integral a product of R integrals. By pulling the sums into the integral, we m" rewrite the above bound in the form s R (4.9) I" j=l i=r+l Thus, to prove Theorem 1.2. it is enough to show that the right-hand side of (4.9) is at least 2. To this end, we give a dissection of the region of integration Il" into three subsets. oughly speakiug, we expect that the main contribution to the integral in (4.9) cornes from the region where the first r colnponeltS of are "close" to rationM lmmbers with slnall denolnilmtors and the lt R - r components of are very small in absolute value. We will shov that the contribution to the integral in (4.9) from this region, the so-called major arcs, is positive and "large", and we will also show that the contribution to the integral froln the other regions is slnaller, and thus the integral over all of Il" is positive. For notational ease, we set 1 (4.m) = 4(n + 1)" We now define, for positive integers q and integral vectors a = (al, a2 ..... af) G Z r, and real nmnbers P with P 2, the region ,M(q, a), or (q. a. P), bv (q,a) = {G[0,1] x [-(logp)Bp-k,(logp)Bp-k]-. i (logp)Bp - for 1 < i < r q (4.11) here I]xl[ denotes the distance from the real number x to the nearest integer. X define the nmjor arcs to be the region («.12) =(P)= (q,,p), lq(log p)B a (mod q) (a ..... a ,q)=l SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2687 where by a (mod q) we mean that a nms over vectors a E Z r such that one has 1 aiq for 1 < i<r. In section 5, we will prove the existence of a functiol T(P), with T(P) k 1 and limp T(P) = oe, and satisfyiug a certain property. The flmction will depend on B and the coefficients of the forms Fi. We define the minor arcs to be the region (4.13) m = re(P) = ([0,1] x [-T(P),T(P)I -) N (P). Finally, we define the trivial arcs to be the set (4.14) t= t(P) = {a e I1 > T(P)}. 5. AN ANALOGUE OF I, VEYL'S INEQUALITY In this section, we give an mmlogue of Weyl's ineqnality. For any rem number T with T _> 1, define the region (5.1) mE=roT(P)= ([0.11 r X [-T.T] R-r) \.Ad(P). We now state the central lenmm of this section. Lemma 5.1. Fix a positive real number T with T > 1. Define the forms Fi(x) as in (1.2) for 1 < i < B. the regio, roT(P) as above, and gj(. P) for 1 j s as in (4.8). Suppose that the coeciet matrix A associated with the system F bas rank R. Suppose also that the irrationality coditio (iii) of Theorem 1.2 holds. Then one bas (5.2) lira sup H= 'gj(.P)' = O. Poe «mT(P) ps Observe that trivially one has H IgJ(a" P)] <- P" so we are only seeking a slight j=l improvenmnt over the trivial bound. We also note that the central ideas of the proof stem from the work of Bentkus and G6tze [3]. In order to prove Lemnm 5.1, we first need to give another lemma, which is essentially a combination of two analogues of Weyl's inequality for exponential sunls over smooth mlmbers. We first quote these two analogues, essentially due to Vaughan and Wooley. as they are presented in [5] as Lenmms 3 and 4. respectively. Lemma 5.2. Let c and P be real numbers with P >_ 2. Define g(c) = g(c. P) as in (4.6). Fix a positive real number e. Then for su]ficiently small q. there is a positive real number3, that depends only on k such that either one bas Ig(a, P)[ <_ p-'r, or there are relatively prime itegers a ad q with q > 1 that satisfy g(a,P) « q¢P (q + pklqa--al) -W(2k) (logp)3. Lemma 5.3. Let c and P be real numbers wzth P > 3. Define g(c 0 = g(c. P) as in (4.6) with 0 < r/< 1/2. Fix positive real numbers A and e. Suppose that a and q are relatively prime integers with 1 <_ q <_ (logP) A and Iqc - ai <_ (logp)Ap -k. Th.en one has g(c,P) «, qP (q + Plqc - al) -W . 2688 ERIC FREEMAN We now state the COlnbinatiol of these lmmnas. Lemlna 5.4. Define 7 = 7(k) as in Lemma 5.2. Fi.r positive real wumbers 0 and B'. Suppoe that P is a real umber with P 3. a,d that It is a al umber with Dee 9() = 9(o, P) as m (4.6), u,ith q s.uciedl small, ad suppose that oe (514) 19(,. P)I ,P. Then lhere ei.sl. a positive itleger q and an ileger a wilh (a. q) = ] and q ççB',k,o I -k-kO and q - al ççB',k,O p-k-kop-. Proof. lt. is «lea.rly enough to aSsulno that we have 0 _< 1/2. \Ve apply Lellnlla 5.2 with the «hoi«e « = 0/(2k). By (5.3) and (5.4), thele exist relatively prime integers ri ami q with q 1 su«h that t)ll(' has I,P 5 19(. P)I « qO/(2} p (q + pklq _ al)-l/(2) (log p)3. It follows that q-e « p--2k(logp)6 and P]qo - a I « qe#-2k(logp)6k. By (5.3) and thc condition 0 1/2, we Cel'tainlv have ('+a) q << (log e) qïç;) ,,« Iq - cri « (log ) -. Now we may apply Lemma 5.3. for large P. choosing A = 5k (B' + 3)/(1 - 0). say. and e = 0/(2k). XX obtain , « ¢/( ( + elv _ 1)-'/ It follows that one has q « p-kqe/2 and pk]q _ ai « p-kqe/2. Thus, since p 1 lnust hold. xve have 2k ]l_k_k O . q « 2-o << and Thus the proof of Lcnmm 5.4 is complete. Now xvc are able to give the proof of Lemma 5.1. Proof. Suppose for the sake of contradiction that the condition (5.2) doe hot hold. Then there exist a positive real number e, an increasing seqnence of positive real mmbers P with lim. P = , and a sequence of real vectors , mr(P) with nmv clearly ume that we bave e < 1. Bv trivial estimates, we have I(a, )l > Now we apply Lemma 5.4 to the sums gj(, P) =g (A)(), P) for ail suffi- cientlv large choices of n. For sufficientlv large t, wc have the bomds ç P, and SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2689 e > (log P,)-, and also P, > 3. Thus we nlay apply Lenmm. 5.4 with p = e and = 1/k and B' = 1. Therefore, there are constants çl and c2 that depend only on k such that for large n an(t for 1 j s, there are integers q,a and anj that satisfy (5.5) 1 qnj C1 -k-1 and [Ai(a)qi -- anj[ C2e-k-lp2k. It follows ffoto these bomds and the definition (5.1) of mT(P) that we bave for ail j with 1 j s, and all large n. For fixed e and T, we thus have that la.il and q,j are uniformly bounded. So there are onlv finitelv manv possible (2s)-tuples ( qnl, qn2 ..... qns, anl, an2 ..... Ons ). Therefore one such (2.s)-tuple, say (ql ..... q.,a ..... a). oc('urs infinitelv often. Thus there is some sul)sequence, say {.f,}, wi/h (q .... , qa,,s, ar,,, .... , (lfims) = (ql,---, q.ç, (il .... , forallmZ +. Since the sequence {ar,} is ('ontai,,ed within the compact set [0.1]x [-T, TI -, t,,«r i fm-thcr subsequence {a,,} and a ve('t,)r a0 e [0.1]x [-T,T] n- such that Our goal in the remainder of the lemma is to show that for suciently large values of m, we bave a,m (P)" which contradicts our original assumption. By (5.5) and the defining propcrty of the subsequence {h}. we have (5.6) IAj(a,)qj-aj<ce--lP - for l<j<s andforall mZ + Taking the limit of both sidcs of (5.6) as m goes to infinity, we obtain (5.7) Aj(a0)= a for 1 j s. qç Because condition (iii) of Theorem 1.2 holds, denoting a0 = (ara. a0,..., a0n) we nmst have (5.8) Therefore we bave O0(r÷l) O0(r÷2) .... = OOR. A,-)(o0) __ __ai for 1 qj Now, by (5.6) and (5.7), we have (5.9) [Aj (c, - = - -- + -- - A(o) < qj qj for 1 j s and for all m N+. Now, because A has full tank, we mav assume by relabeling variables if necessary that the submatrix A, defined as in (3.2). is nonsingular. Because of this and because the bound (5.9) hokls, in particular. for 1 < j < R, we must have -0 < c3(F)e -k-lP-k for some constant c3 = c3(F) and for all m N+. Therefore, by (5.8). we must have .10 , e [0.] [_--l2,î,--,p-- o ,,, e z+. 2690 ERIC FREEMAN If r = 0 holds, then for m suiïicientlv large, onenmst have ,m E M(Pn). But this contradicts our original assulnptiol that the sequence , satisfies , G my(P,). whence the equality (5.2) must hold. So we may assume for the remainder of the proof that 0 < r R holds. Then, using (5.6) and (5.10), for m ff Z + and for 1 j s, we have A(.r)/ )_ al Aj(n) ad ija.ni (( e_k_lp_ k (5.11) 3 'tre qJ qJ Since «41 iS nonsingular, there is an r x r SUblnatrix, say A0, of A that is nonsingu- la.t. assume for ese of notation that Ao is the upper left-hand r x r subnlatrix of AI, noting that the other cses ail follow in the saine fashion as this case. For any real vector a = (al,... ,an), write a' = (a ..... a,.). By (5.11), we bave ol/ql [ U'ml T a2/Ç2 Or/qr tt'mr fbr some real vector w, = (U'l. tt',) with Iwl EE e-k-lP -k Since we have assulned that A0 is nonsingular, we mav use Calner's rule to find b = (bi ..... br) with a/ql ] T a2/q2 A 0 b = ar/qr Since A h integral entries, Olm moEv see that bi has the forln b = di/q for i < i < r, where dt is an integer, and q is a positive integer that satisfies (5.12) q (qlq...q)det.(A) c4(F)e -(+1, where the last bound follows from (5.5). moEv aSSUlne, by reducing if necessary, that we have (dt, d2 ..... d. q) = I. By Cramer's rule again, we Inay find v N with Av = wm, where we bave (s.a) lvml «(V) -- Xit.e d = (dt, .... d.), and if d = for sonm i. define di to be q instead. Then we have ,' dq +v (modl) for mZ +. Now fix any choice of m large enough so that we bave (5.14) (og ,) lnaX(«a(),«4{),()) -+1). Then by (5.10) we have G [0,1] x [-(log P)P -k., (log P)P-k] -= for this choice of m. Now write = (d, d2 ..... d, O. 0 ..... 0), where there are R-r zeros here, and define ., similarly. Then, setting u = + (0,0 ..... 0, a.{+x) ..... an), we have a=*d+u (modl), q S5 STEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2691 where (dl, d2 ..... d,, q) = 1 and where, bv colnbilfil,g (,5.14) with (5.10), (5.12) and (5.13), we have l_<q_<(logP)B and u] (logP) . Thus recalling dcfinition (&ll), wehave , , (qd, P) for out particular choice of and in fact recalling (.12), we also bave .(P). As in the case r = 0, this is a contradiction, whencethe equality (5.2) must in fact hold. This completes the proof of Lemma 5.1. At this point, we make an observation about the lemma for those familiar with earlier argmnents of this type. VVe note that in previous work by the author ([10], [11]), the analogue of our Lemma 5.1 was proved with two different methods, for two subregions of the region roT(P). If we were to proceed by analog. with earlier arguments, we would instead have to treat a region mT, To (P), say, in place of mr(P), for positive rem numbers T0 with T0 _< T. The new region would be defined by mT,To(P)=mT(P) C {O : ]0] >_ T0}. Essentially by combining the arguments used for each region in previous proofs, we are able to dispense with the reqlfirement lai > T0. Having done most of the work, we tan now give a lemma that essentially savs that H IgJ(a" P)I is small for a E m. The idea of using such a lenmla is due j=l originally to Bentkus and G6tze [3]. Lemma 5.5. Deflne the forms Fi(x) as in (1.2) for 1 < i < R and the exponential suns gj(. P) for 1 <_ j <_ s as in (4.8), with r i sufjïciently small. Suppose that the coefjïciet matrix A associated with t]e sgstem F bas ratk R. Suppose also that the irrationalitg condition (iii) of Theorem 1.2 ]olds. Then tbere exists a function T(P) that depends only on B.! ad the coefjïcients of the forms F. F2 ..... F, that satisfles T( P) >_ 1 and (5.15) lira T(P) = P-cc and such that if we define re(P) as in (4.13) with this choice ofT(P), then one bas sup H Igj(,P)l : o(P) . om(P) 2î-: Pro@ The lemma is very similar to Lemma 6 of [10] and Lelnma 4 of [11], and the proof follows iii a silnilar fashion. [] We note that this lelmna (and Lemlna 5.1) holds for a.ny positive choice of B, but that the function above that is o (P) depends on B. We have stated this lennna in a general fashion in the hopes that it nmy be useful for future workers. We observe that one could ensure that the function that is o (P») depends only on B, r] and 2R - r of the coeflïcients. This follows, with some effort, after finding a subset O r C_ {1,2 ..... s} with IJI --= 2R - r such that the conditions Aj(o) ( Q for j J, taken together, imply that c+l ..... cu = 0. This can be proved, although out method of proof, at least, is hot straightforward. 2692 ERIC FREEMAN Iu the remainder of the paper, we fixa flmction T(P) that satisfies the conclu- sions of the above lelnma. We note that this is the special function we referred to above in section 4, and is used to define the lninor arcs and trivial arcs. We observe at this point that we could obtain corresponding results which are very similar to Lemmas 5.1 and 5.5 if the expouential sums gj were replaced by exponential sums over a complete int.erval. The only major change needed would be to use Lemma 2 of [11] in place of out Lelmua 5.4. O. THE lk|INOR ARCS In this section, our goal is to show that the coutribution from the minor arcs to the integral in (4.9) is o (P-nk). We first give a lemma, which is essentiallv a restatement of results due to Vaughan [19]. [20], and results due to Wooley [23]. Lemma 6.1. S-uppose that k is an nteger with k >_ 2. Define g(c) as in (4.6), with q sufficiently small. Then there is an absol-ute positive constant ff' such that ff t is a real number satisfying either (i) t >_ miu (2 , L'(log/," + h»g log k + 2) + \ OF then one ha C'k log log k ' for k > 3. log k J - (il) t > 4 for k= 2. (6.1) ff01 ig(c01 t , pt-e. We observe that one could certainlv improve Oll the lemma iii certain cases, but we choose to use only the above bounds for out results. Pro@ If the first bouud of coudition (i) holds, then the result is Lenmm 6 of [11], which is essentiallv due to Vaughan [19]. [20]. If on the other hand. the second bound of condition (i) holds, then we may essentially quote Lenmm 7 of [11], which itself follows ahnost immediatelv from work of Wooley [23]. We note that the 3 in Lemma 7 of [11] has been replaced bv a 2 here; I aih gratefifl to Scott Parsell for showing me the technique one uses to make this inlprovement. In the case in «hich (ii) holds, we give a proof for completeness. Define e=t-4. VCe need only prove that Olle has [g(a)[4+«d a « p2+«. Clearly. we may assmne that e _< 1 holds. For convenience, we write 2 Defiue 91 = rl ° G [0.1]'lg(cOI > P(log p)-C;}. Also, for positive integers m. define S_ STEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2693 Now for o E -qlm, we apply Lenmm 5.4 with the choice/? = G. Thus, for large P and any positive real number , there exist coprime integers a and q with q _> 1. and << 2 m(2+6) and ( - î << q-12m(2+6)P-2. q q Thus ,are have It follows fol" < e/2 that On the other hand. one has (.3) But ri0.11 q q2rn2+5) a---1 « p2+e 2-m(-26) « p2+e. I9(a,)lada is less than or equal to the nmnber of sohltions of the equation with 1 _< :ri _< P fol" 1 < i < 4. This is bounded bv a constant nlultiple of p2 log P, a well-known result, which can be proved bv elementarv means. Thus ff'oto (6.3) and the definition of 9l. we have [0 Lq(°)[4+¢d (P(log P)-) p2 log P « P:+¢ (log p)-l, by our choice of G. Combining this bound with (6.2) completes the proof of Lenmm 6.1. [] We note that (6.1) is an example of what one might call an "'exact Hua inequal- ity". In most work using tlle Hardy-Littlewood method, one uses bounds of the type (6.1) where one onh" needs to show that fol" anv « > 0, the left side of (6.1) can be bounded bv pt-k+¢. Dispensing with this e is crucial for our work. The use of such an inequality stems from the work of Bentkus and GStze [3]. For the renlainder of the paper, we now fix a choice of so that Lemmas 5.5 and 6.1 hold fol" this choice. Now we turn to what is essentiallv our analogue of Hua's inequality. It is very similar to Lemma 8 of [10]. Lenmm 6.2. There is an absol,de positive real constater C_ with the followin9 property: Assume that the forms F1. F2 ..... FI are as in Theorern 1.2. with coejïjïciet matrix A satisfyig (3.2) and (3.3). Assume lhat f is a positive integer satisfging (>_5 for k=2. and ( 72k l°g l°g k )fork>3. g >_ nfin 2 + 1. k(log k + log log k + 2) + log t,- - 2694 ERIC FREEMAN Define the exponential smns gj(o, t 9) as in (4.8), and deflne the function K as in (4.2). Let d(P) be a nonnegative real«,alued.funetion, and let n be any subset of the region Also, define Proof. Observe first that for an3" real number e with 0 < e < 1, one has s R L II 'gJ(«'P)' 1-I j=l z=r+l s « sup lgj(. P)l IK(ai)[da. It follows fFOlll trivial estiinates that Olle has R j=l i:r+l « (h(n" P)P») (P»-gn)- L eR R H IgJ(«" P)II-e H 3:1 i=r+l We may certailfiy choose a positive real number e so that we have (6.4) g(1-e)>4 if k=2. Defining C' as iii Lenuna 6.1 and choosing OE2 to be sufficiently large, we lllay ensure t hat we have ( C'kloglogk ) .t _> lnin 2 k + 1. k (log k + log log k + 2) + log k + 1 if k _> 3. Thus we ma.v choose a positive real nulnber e, slnall enough (in terlns only of k and C) so that we have ( C'k log log k) (6.5) g(1 - e) > min 2 . k (log k + log log k + 2) + log k if k > 3. In each of the cases, we denote our particular choice of e by u. Now one can join the proof of Lemnm 8 of [10] after equation (66), and then follow the remainder of that proof with only slight adjustments. The bounds (6.4) and (6.5) are the crucial bounds that, we need to apply Lemnm 6.1. V'e omit the details. [] Now we can wrap up our work on the minor arcs. We have the following lenmm. SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2695 Lemma 6.3. Suppose that we are in the sctting of Theorem 1.2 and that the coejïficiet matrix A of the system F satisfles the corditions (3.2) and (3.3). Choose a function T(P) as ir Lemma (5.53. Define the exponertial sunrs gj(o) as in (4.83, with sufficiently srnall, the region m as in (4.133, ard the function Ix" as in (4.23. Then one bas s R Jm = "3(0)I i=r-t-lH I'(°zi)[d°z =o (tgs-'k). Pro@ We siinply apply Lenuna 6.2 with the choices n = m and d(P) = 0. VVe ha.ve h(m, P) = o(1) by Lelnma 5.5. Tlms the proof of Lenllna 6.3 is coniplete. [] 7. THE TRIVIAL ARcs Iii this section we show that the contribution frolll the trivial ai'es to the integral in (4.9) is o (ps-R), which is now easy to do, having done the necessary work above. We have the following lenlnia. Lemma 7.1. Suppose that we are ir the setting of Theorem 1.2 ad that the coejïficiet matriz A of the system F satisfies (3.2) and (3.33. Choose a functior T(P) as in Lemrna (5.5). Defl, e the exponential sums gj() as in (4.8) with q suciently small, the region t as in (4.143. ad the functio Iç as in (4.2). Then one has s R Pro@ YVe apply Lenuna 6.2 with tlie choices n = t and d(P) = T(P). h(m, P) = O(1) by trivial estiinates. Thus we obtain R fil IgJ(°t)l H IIx-(°q)'d° «7. (T(P))-IP s-Rk, = i=r÷l which by (5.15) of Lemma 5.5 is o(PS-Rk). complete. We have Thus the proof of Lemma 7.1 is 8. THE IklAJOR Aacs We now treat the lnajor arcs. Our goal is to show that for large P we have s R 3----1 i=r÷l 8.1. Approximation on the major arcs. We start our treatment of the major arcs by approximating the functions gj(o) by auxiliary functions. We need soine notation before we do so. We define Dickman's function p by the conditions (8.1) p(,) =0 p(u) ---- 1 up'(u)-----p(u- 13 p is continuous pis difïerentiable for u_<0, for0<u< 1, for u > 1, for u >0, for u > 1. 2696 EIRIC FREEMAN Also, for real nulnbers ff, define the functiol fo Pk Ç log_x_ 1 oc(l/l)_lp e(:r)dr. (8.2) () = » og P ] For real vectors Nn, define Aj() in (4.7), and write (8.3) wj() = w (Aj()) for 1 j s. Also, for integers q and a with q k 1, define = k q We now collect sonm results, given by Brfidern and Cook [5], in the following 1PIllllla. Lelnma 8.1. Defit, e 9() as i (4.6) and ([) as i (8.2). Suppose ha¢ a and q are iegers with q 1. ad ha is a real number. The one bas and oe(fl) « lnin (P, Proof. The first result is simply equation (29) of [5]. The second result is essentially the third centered equation on page 135 of [5]. [] We note that, as remarked by Brfidern and Cook, a and q are hot required to be relatively prime. We now state the central lemma of the section, which is ver3" similar to Lemma 4.4 of [16]. Lemma 8.2. Suppose that we are in the setting of Theorern 1.2 and that the coefficient rnatriz A of the system F satisfies (3.2) and (3.3), and that there is a real vector z satisfying (3.6). Define the so-called singular series by (3 = 1 r = 0, and (8.5) @:Z Z q-S H S (q, AJr)(a) ) for q=l a (nod q) j=l (a ..... a,.,q)=l and the singular integral Z(P) by Fiz any positive real number e. Then if P is a sufficiently large positive real number. oe bave « p-n," ((log p),(+,)- + (logP),(-(e/k)+,)). SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2697 PTvqf. There are three steps of the proof. One first at)proximates each fimction bv terres of the form q-lS (q, A.T)(a))coj(/)on each of the major arcs, then gj( o) one extends the integration over each nm.jor arc to ail of IR R, and then one extends the smn over q to all positive integers q. The argument closely follows the proof of Lemma 4.4 of [16]. One major difference involves the use of the approximations given in Lemma 8.1. Since we are using exponential smns over smooth numiwrs, we need to use these approximations instead of mre standard results for exponential sums over complete intervals. Finally, we observe that the only condition we really need on f for the purposes of this lenuna is the b,mnd f >_ 2/,: + 1, [] Now we turn to consideration of the singular series and the singulm" integral Z(P). In particular, we shall show that we have 9 » 1 and also that xve have Z(P) » P-Rk for sufficiently large P. We first treat the singulm" series. 8.2. The singular series. \'e give some (lefiuitions. Sui»pose that G1, G2 ..... Gr are r integral diagonal fOllllS iii s vm'iables with coefficient matrix B, with entries dij. hl Section 8.2, we consider integral forms, and of coin'se we aSSllllle that r _> 1 holds throughout this section. \Ve then define the singular series (G} associat.ed with this svstem of r finms by (8.7) where we set e=e(.¢)= E q-* H S(q, ab q=l a mod q) (al ..... a,.,q)=l r aba)=Edijai for 1 <_j <_s. "We return to the notation of the rest of the pat»er for a momcnt. Observe that the definition of given here coincides with the dcfinition (8.5) in the case r = R. Moreover, in general, whcn 1 ç r 5 R hokls, the first singular series is exactly the latter singular series, where the latter is associated with the first r forms F1 ..... F. Note also that the first singular series is independent of the R- r fonns Fr+l ..... F. Suppose that p is a prime and , is a positive integer. Then we sav that an integral vector x = (1, x2 .... , ,) is a solution of rank r (mod p) of the svstem of congruences (8.8) Gl(x) if there is a subset J ç {1, 2 ..... s} with ]JI = r such that one has p det(Bj) and p { x. Also, for anv prime p and any positive integer n, we define M (p, G) to jJ be the number of solutions x (mod p) of the system (8.8). Now we shall define the concept of a nonnalized systeln of forms. follow Low, Pitman and blff [13] closely, but we need a slightly more general notion. essentially want to define a notion of a system such that a related system, which results after setting all but some subset of tf of the variables equal to zero. is normalized in the original sense of Low, Pitman and kXlff. 2698 ERIC FREEMAN Suppose that the coetï-icient matrix B contains t disjoint nonsingular r x r subma- trices BI, B2,..., Bt. To be clear, t)v this we lnean that there is solne permutation of the columns of B so that the first r columns form a nonsingular submatrix, the second r cohnnlls form a nonsingular sut)nmtrix, and so on, through the t th set of r cohmms. We define Now let j = (j, j2 ..... jt) t)e the ordere(l (tr)-tuple su«h that the particular matrix [B1B2... Bt] is the sut)matrix of B consisting of the columns of B indexed in order t)v j,j .... ,j; that is, we define j so that for 1 v t and 1 b r. the -th cohmm ]t,(r-l)+h dcpcnds on j. Also set . = {j. j,..., j,}. Suppose that p is a prime dividing A. Here we define, following [13] closely, a p-operation on the fornls G1 ..... G r aS a transformation that produces integral forns H .... , H, and has the fidlowing steps: (i) Pre-multiply B by an integral unimodular matrix U with entries in the set {0. ..... (il) Next, multiply at most tf" - t" of the cohuims of U.I by pk and multiply anv of the colunms of U{1 ..... s}kJ by pk; (iii) Then divide g of the rows by p. where we have 1 N g r. As discussed in [13], step (i) corresponds to adding linear combinations of some of the forms to one or more of the other forms. Step (ii), on the other hand. corresponds to writing xj = pgj in each cohuun j that one multiplies by pk, and then trying to solve the new inequalities in the variables yj. Step (iii) corresponds to dividing g of the r equations by p. One can check. in [13]. that a p-operation is possible for all primes p that divide Note that for the resulting system H. we bave for sonle ilteger m. say that such a p-operation is permissible if one h m<0. Observe that. upon perfornling permissible p-operations for anv of the primes p dividing A(G), we can find a system H that can be obtained from the original system G via a finite sequence of permissible p-operations and such that A(H is minimal. If G is a system of r integral forms as above, such that A(G) cannot be reduced by any permissible operations, then we say that G is a (j. t)-norlnalized system. Finally, if G is a system of r integral forms in exactly tr variables, then we simply say that G is a normalized system. note that. in this case, our definition clearly agrees with the definition given for a normalized system in [13]. e make one other observation. Suppose that. above, the coecient matrix B of a system G, in s variables with coeNcients dij, contains t disjoint nonsingular r x r sublnatrices B, B,..., Bt, and define j and J as above. Then. we define the system G* in tf variables, by defining, for y Z t and for 1 < i < r. the forms (8.9) SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2699 Note that this is simply the system obtained bv setting ail variables with indices j J equal to 0, and subsequently reordering the variables. Observe that the coeffiçient matrix B*, sa)-, of the system G* has the form B*=[ B, B2 Bt ]. Suppose nov« that a system H can be obtained rioto the system G airer a finite sequence of permissible p-operations, and suppose that H is (j. t)-normalized. Then consider the system H*, defined as in (8.9). We can sec that the saine p-operations (restricted to the columns j E J) allov« one to obtain H' from the system G*; airer all, the variables of G* are a subset of those that appear in G. Each operation is certainly still permissible, since the definition of A involves only the coluinns j E J. If one could reduce A eH') via a permissible p-operation, then we could simply extend step (ii) and nmltiply all of the cohmms of UB{ ..... s}\J by p#. This would give a permissible p-operation for the system H, which contradicts our assmnption that H is (j. t)-normalized. So if H is (j. t)-normalized, tllen it follows that H' is normalized, and moreover, if H results from G airer a finite sequcn('e of permissil»le p-operations, then H' results from G" ri'oto the saine sequence of p-operations. We can now sIate the following lemma, which is a step towards bomding the singular series below. We do slightly more than we need to, in the hope that it vill be usefifl for fiture workers. For this reason, we state I he lemma in a self-contained lllallller. Lemma 8.3. Svppo.e that r, k and s are positive integers with k >_ 2. and suppo.e for l < i < r that D,(x) : dilfl" Î ÷ d,22" +... + dis.r fs an integral diagonal form of degree k. Suppose that s satsfles s k tr. where t fs an iteger satisfying t>2k+l. Suppose that the coecient matrix C of the system D contais t disjoint nosin9ular r r submatrices C1, C2 ..... Ct. and deflne t A(D) = Also, define the sig.ular series (D) as m (8.7), Suppose that the followmg prop- erty, which we denote by Pet, k, r), holds: Given ang system of r itegral diagoml forons G1. G2 ..... Gr of degree k in tr variables, with coeciet atrix B which consists of t disjoint osingular r r submatrices and such that the system G is ormalized. then for every prime p and every positive integer n. there is a solution x of tank r (mod p) of the system of congrueces Gl(x) G2(x) --- oeGr(x) 0 (nlodpn). Then the series (D) converges absolutely and one bas (D) >>D 1. If one also bas t>kr+k+l, Proof. We first give some more notation. For anv prime p, define 3' = "(k.p) bv «hoosing - to satisfy p'-II/,', and settiug { 1 if T=0 -) = "r+l if 7->0andp>2 T+2 if "r>0andp=2. Also. for anv l)rime p. we definc (8-J 1)) \D(P) ---- .'il (p'. D) __,oe pn(s-r) As iii Q h«l|)t( r 5 of [61, and ll,'qing al,,qo Lenllna 2.1() of [15] OllC lllV see that this limit exists, that (D) converges al)solutely, and that @(D) is equal to an absolutely ('(llV('l'gellt l)r()(luct, that is. we have (.) (D) = H P (XXb noie that it is in his argument that one uscs the condition t k 2k + 1. and that he rate at which the product converges deI)Cnds on D.) Now dcfine D as in (8.9), and dcfine j and J as in the discussion above. may find a (j. t)-normalized svstem G. which can be obtained from D alier a finite sequence of I)ermissible p-operations. As we bave noted above, G* is then a nor- malized system, which ont obtains from D* aftcr (essentially) the saine permissible p-operations. Since proper/y P(t. k. r) holds, there is, for ail p and -n, a solution w of rank r (mod p) of the svstem of congruences G*(w) 0 (mod p). Bv the way G* w dcfiued, one can s that if y = (y, ye ..... y) is defined bv { wj if j J .v = o if j C j. /hen we have that y is a solution of tank r (mod p) of the svstem of congruences G(y) oe 0 (mod p). In t)articular, this holds for n = 7- XX mav thus apt)ly Lelnma 6 of [131 to the system G. whence we have (8.12) 5I (p,G) k p(»-)(-) tbr n > 7- Froln this fact we will deduce a lower bound for 5I (p, D). To this end. suppose for some positive integer r tha.t y Z is a solution of the COllgrHellCeS Gi(y) Ç2(y) ""Ç,-(Y) 0 (mod pn). Recall that the svstem G resulted from D after a finite sequence of permissible q- operations. Let H be a svstem such that G arises from H after a single permissible q-operation. Let I be the subset of {1, 2 ..... s} consisting of the cohmms affected bv step (ii) of this q-ot)eration, that is, let I consist of the indices such that the SYSTEIVIS OF DIAGONAL DIOPHANTINE INEQUALITIES corresponding cohmms in step (ii) are nmltilflied by q. x = (Xl,X2,...,a's) by setting We show that one has { qyj if j C I a if jI. 2701 Then define the vector (8.13) H, (x) = Ha(x) =.-- --- Hr(x) _= 0 (mod To see this, let H (0 and H (i) i,e the syst.ems that result a.fter steI)S (i) and (ii) of the q-operation, respectively. We certainly have H¢ii)(y) = 0 (mod t/): indeed. sonle of these forlns are congruelt to 0 (lnod pq). Then observe that we bave H(ii)(y) = H(i)(x), whence H(i)(x) 0 (lnod p) holds. Since the lnatrix U is unilnodular, so that in particular its deterlninant is hot divisible by p. one has that (8.13) holds. Thus any solution y of G(y) 0 (mod p) gives lise to a solution x of H(x) 0 (mod p). If q p holds, we therefore have M (p",H) k Al (I,',G). If q = p holds, we might have some reduction in the nulnber of solutions, because lnultiplicatiol by p in /pn has kerlml of size p. but we certainlv bave M (p", H ) So, by repeating this analvsis for each permissible q-operation, one can see that if G is a (j. t)-normalized system arising from the svstem F after a finite sequen«e of permissible q-operations, then one has M(p,D) OE if p'[[(D). pro(s-r) Now the linfit k(P) exists; this follows in nmch the saine wav as the corresponding fac for D. It follows that we bave (8,1) k(P) p_) if pH(D). It follows from (8.12) and the definition of k(P) tiret for ail primes p we bave (8.15) kD(P) p-(v+ord(aD)))(-). Since the product kD(P) is absolutelv convergent, there is a constant c(D), which mav depend on D, such that we have 1 Thus, using also (8.15). we have 1 1 pNc(D) pNc(D) 1 (A(D)) - p-(- »D1. pSt(D) 2702 ERIC FREEMAN Now suppose that we have t >_ kr + k + 1. One can prove as iii chapter 5 of [6] that for primes p, one has (8.16) n=l where for positive integers , we define s (p') = Z H S(p''Mj(a))" a (mod p') j=l (al ,...,a,p)=l Now suppose that p{ A(D). Sut)pose for 1 _ v t that Cv consists of the cohmms j,,jv .... ,j, in that order. Then for a satisfying (al, .... a,.,p) = 1, there nmst exist sonle j {j, ..... j. such that p ]iii(a). since we have p { det(C). Thus for any prime p with p A(D), and any positive integer , by the standard estimatc S (p', a) « p,(1-(1/¢)), which holds for (,p) = 1, we have S (p') << p- Z pnS-(nt/¢) « pn(r-(tl¢)). a (mod (al ..... Combining this last bound with (8.16) and using t >_ kr + k + 1 yields \D(P)- i EE Zp nr-(t/k)) EEp -1-(1/k) for p{A(D). n=l So there is a constant C that depends only on k, r and t such that one has (8.17) I\D(P) - 11 < CP --(/) for p{A(D). Now, because Z Cp-I-(1/k) converges, there is a constant C that depends onlv p on k, r and t such that one has 1 - C -1-(1/t¢) > 0 for p > C and 1 H (1- Cp -1-(1/k)) _ . p> Now for all p we have \D(P) > 0 ff'oto (8.15); so by (8.17) we have plA(D) p'A(D) plA(D) pA(D) 1 I-[ (1 I-[ D()" p[A(D) SïSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2703 It follows from (8.15) that we have 1 p p<(, p/XID) H p--('+ord,,( A(D) ) )(s--r) (A(D))T-* H P-«-) plA(D} (A(D)) '-* (A(D))20 "-) " This completes the proof of Lemma 8.3. Now we give another lemma that builds on the above lemma and completes the treatment of the singular series for the cases k _> 3. As in the case of Lemnm 8.3. we do slightly more than what we will need, and we stale the lemlna in a self-contained fashion. Lemma 8.4. Suppose that r, k, and s are positive integers, ad suppose for 1 <_ i < r that Fi(x) = Aixî + Ai2x2 +... + Aixs is an integral diagonal form of degree k. Suppose that the coefficient matrix 4 of the syste F contains g diooint nonsingular r x r submatrices .4 ) A () AW) where g is a positive iv teger satisfgig fk2k+l. « = H ]dot (A}:')]. Define = (F)as in (8.7). Finally, suppose that one of the two followin 9 statements holds. (i) k is odd, and k 3 holds, and one bas g kmo, where mo is the least positive integer m such that one bas (ii) k 2 3 holds, and one bas g > k [48k = log 3rk =] . Then oue has O(F) » 1. Moreover. if g>kr+k+l holds, then there exists a positive real constant c(k, r, s) that depends only o k, r and s such that one bas rg(F) _> c(k, r, s)A -. Moreover, we note that if k is odd with k >_ 3. then there exists a absolute positive real constant C such that condition (i) holds if one has rk log 2/," (' > + Ck log(r log 2k). - log 2 2704 ERIC FREEMAN Proo.f. The last statement of the lemma can be checked with a straightforward computation. Thus to prove the lemnm, we need only check that the condition P(& k.r) of Lemma 8.3 holds for out choices of (. So suppose that G is a normalized system in l'r variables. For the case in whi«h condition (i) holds, we may apply Theorem l(ii) of [13] to see that for any positive integel" 71, the svstem G(x) = 0 (mod p') has a solution x of tank r (mod p). On the other hand. for the case in which condition (ii) holds, we may apply Theoreln 3(i) of [13]. Tlms in either case, the condition P(& k, r) of Lemma 8.3 holds, v«hence Lelnma 8.-1 follows. Now wc give a lclmna to treat the singular series in the case k = 2. \Ve observe that one could surely obtain a rcsult that is better for large r, but we choose hot to pursue this hcre. Lemlna 8.5. Suppose lhat r and are positi,e bdeger.s, and suppose for 1 < i < r that Fi(x) = )il.Fî -- )i2J'72 --...-- i.s an itegral diagonal quadratic form. Suppose that the coeJficient matrix A. defined as in (1.3). contams g disjoint 71oT,s'ilgular r × r submatrices, where is an integer sati.sfying t' OE lllill (-11.2 + 41" + l. 38-1 log 161" + 5). Define t = t(F) as i'n (8.7). Then one bas 5(F) »F 1. Pro@ If the first bound for holds, then it follows that any nontrivial complex liniar combilation of the forlns F ..... F bas rank at least 4r 2 + 4r + 1. Bv the theorem in [18], the singular series t is positive and depends onlv on the forms F1, F2,.... F,-. One can readilv check tllat the singular series t is defined in [18] in the saine ruminer as we have defined it. Sul)l)ose instead that the second bound for holds. \Ve give a sketch of the proof in this case. We first seek an analogue of Lemma 12 of [13] for the case k = 2, with m_>[1921og16r+2]. For primespwithp<_81.2 , say. it is easv to check that Lenmm. 5 of [7] provides an analogue of the desired type. To obtain an appropriate analogue of Lelnma 12 of [13] for primes p > 8r 2, one applies an adaptation of Theorem 2 of [14], «ith, say, c = 4: one can check that if one assumes that the matrix of coefficients contains c + 1 nonsingular i" × r submatrices, rather than assuming that the matrix is highly nonsingular, then the result still holds. (This can be seen by noting that the inequality q(B) > ci, which would still hold, is the key condition needed on page 339 of [14].) In either case, we bave an analogue of Lemma 12 of [13]. and thus ce can show that condition P(tç, k, r) holds, a.s in the proof of Theorem l(ii) of [13]. [] Michael Knapp and Professor Wooley provided me with a proof of a result closel3 related to the second part of thc above lemlna, for which I alll grateflfl. SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2705 8.3. The singular integral. F{ecall that in (8.6) we defined the singular integral Z(P) -(P) = . Çj=IIlt'UJ(fl) ) (i=rl[Ç(Çi)) d" Our goal in this section is to demonstrate thN for large positive P we have the bound (8.18) g(P) »F P-- Instead of using the traditional approach which uses Fourier's Integral Thcorem, we use a method given by Schmidt [18]. Below we [ollow parts of [18] very closely. Much as in [18], for anv positive rem mmbcr T and anv rem numbers a and , we define 2 lçT(O)=lç(aY 1)= ( sill ( -1) ) l.lld (8.19) (8.o) We now define = j" T(1-Tlfll) for Ifil-<T-1 ¢'T ( fl ) 0 for [fl[ > T -1 From (4.3), one may readily deduce the %llowing identity, which holds for all rem nmnlmrs , namely, @T(fl) = [" e(al3)Kr(a)d«. By (4.4) and a similar bound for tçT(a), the integral converges absolutely for each choice of P and T. We shall see that for fixed P, we bave liln ZT(P) = Z(P), and we will also show that for large T. we have ZT(P) »F p-Rk. These two facts together establish the bound (8.18). To prove the first fact, we give a bound for the difference ZT(P) -- Z(P). Lemma 8.6. Sppose that T ad P are positive real numbers with P >_ 1. Sppose that R. r. I; and s are bte9ers with R >_ 1.0 < r < R ad k >_ 2. S'uppose for 1 < i < R that Fi(X) = ,il,Z'Î ÷ ,i22F22 ÷... ÷ is a real diagonal form of degree k and that for 1 < i < r, the form Fi is itegral. Asstrne that one bas IIF]] >_ 1. &tppose also that the coeJficient matrix A of the systern F is as in, (3.2) ad satisfies (3.3). where one bas g,>k+l. Define Z(P) and ZT(P) as in (8.6) a«d (8.21), respectivel. Then ote bas 2706 ERIC FR EEMAN We note that the implicit coTstont in Vinogradov's notation bere depends af most on H. r. k and s and, in particular, does hot depend on the coeciets of F. Pro@ Observe first that, in the case r = 0, we havc Z(P) = ZT(P). So we can assume that we have r 1. It follows from the definitions (8.6) and (8.21) that we have From the pemfltimate centered equa.tion on page 305 of [18], one has 1--HIçT()«T -e max [e«T-el] e for 1[ <T. i=1 F and for ]] OE T, one clearly has lç() (( 1. çoml,ining these 1)ounds with the estimatc for w() given in Lemma 8.1, and thc 1)(rond (4.4) [or Iç(o). one h Consider the first integral on the right-hand side of (8.22). By H61der's inequality, one has (8.23) j=l « H ,<T ]/]2 H lllill(P']AJ(t)l-l'k) gdt v=l j=(v--1)lR + l Fol" a fixed choice of v with 1 < v _< & one Inakes the change of variable "7 = (71 .... ,TR) = U,,(/3) given by 7j = A(,,-UR+j(/3) for 1 _< j _< R. and obtains j=(v--1)R÷l R 'u-l(') 2 H 111i11 (P, Ij[-1/k) ' 3=1 If "7 = U. (/3), then one has l'ri = IUv()l < IIFll- SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2707 and by Cramer's rule and Hadamard's inequality, one has (8.24) It follow8 that we lmve (8.25) AI' But one has SI-ri< RIIF[[T d7 whence we have R j=l since we have 7 > k + 1 and IIFIIT _> 1. Thus, by (8.22), (8.23) and (8.25), we have 2c(P)- 2eT(P) << IIFII2ST-11p s-sk 1-I A73/g + Ps-Rgf, O H rein (P, IAj(fl)] -llk) dfl. I>_Tj=t 2708 EHIC FREEMAN By Hiilder's inequality and a change of variable as al»ove, we have :Z(P)- :Zr(P) << IIFll2"T-1/"P s-' H AÇa/e ,(/,. r=l I'l--cFa'T j=l where CF,v is a positive constant which by (8.24) we lnav defilm by «e,v = RR/=IIFI]R_ l- d7 since we have t e >_ k + 1. Combining the last txvo bounds completes the proof of Lclllllla 8.6. [] Now we prove a lemlna which states that for T a.nd P sufficientlv large, the quantity ZT(P) is bounded below. Lemma 8.7. Suppose that we are ir tbe selti9 of Tbeorem 1.2 and that tbe coefficient rnatriz A of the sgstern F satisfies (3.2) and (3.3). and that there is a real vector z satisfying (3.6). Suppose also that one has IIF[[ > 1. Define At as in (3.3). Suppose that T aM P are real vurnbers satisfying T > 1 and (12 )1/ (8.26) P > -'n/?2R]lF[[2/lnax(A-2,1 ) where 6 is as in (3.6). Define 2-r(P) as in (8.21). Then there is a constant ci = q(F,k, R.r,s, 5. z) that does hot &pend on T or P such that one bas çT(P) OE Cl Ps-Rk. Pro@ Recalling the definition (8.3) of wj(13), we can write the absolutelv conver- ,=i)1 ( og« r t=l i=r+I SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2709 Using the identities (8.20) and (4.3), raid recalling the definition (3.4) of Li(x), we can rewrite ZT(P) as 1 )l/k-1 IïI p \klogP - ,pkls(.rl-.-Xs )I*',T(Li(x)) 1Ç[ g'(Li(x))dx. j=l t=l i=r+l ZT(P) is certainly larger than the corresponding integral over the slnaller region __.pk , noting froln Lelnlna 12.1(i) of [21], alM (8.1), that the functiol p is always nonnegative. Fol P satisfying (8.26), one certain|v has P > (5/2) -l/k, 6pk -- < xj < pk, it follows ff'oto (8.1) that one has whence for xj satisfying 2 - - Ç log Xj " ] = - So we have z.(P) > ,;. .] Now define T4p.T to be the region min\l'A//'() fol'l</<," and 37" ( ) } IL(x)l < mil, 1.A'l/ fort + 1 <z < R It follows froln the definitions (8.19) and (4.3) of vr(a) and b(a), respcctiveh, that we have (8.2/) T(P) 2> ps-skTr]t s (TPp.T), where Ps denotes s-dilnensiolml Euclidean lneasure. We now lnake the linear clmnge of variable w = V(x) given bv Lj(x) for 1 _< j <_ R w j= .r for R+l <_j <_s. Since A1 is nonzero, we can see that we have (8.28) /£s (7-P,T) >F //s where we define $.7 to be the region V(74p.T). Note that $ T is the set of w [î ] such that there exists an x 6 , Pk with w = l(x) and such that one has lnin(1./N}/R) lnin(1,zll/R) I**'1 < 3T fol. 1 < i < r and Iwl < 3 for r+l < < R. Now we give a lenuna, which is essentially due to Nadesalingaln and Pitlnan. (SeP [16], Lellllll& 5.2.) 2710 ERIC FREEMAN Lemma 8.8. Let R and s be positive integers satisfying s > R. Let A = be a real R x s matrix. For 1 Let A dcnotc the absolu.te value of the detetinant of the lefl-hand R x R subatrix of A. Suppose that we bave A > O. Additionallg. suppose that Q is a real number satisqing (8.29) Q Suppoe also that w,..., wn are çeal number satisfying ]wl< for l < i < R. - 3 Let 5o = So(W,..., wn) be th« set of all real vectors (yn+ ..... y) [-Q. Q]Æ-n for which there exist real muber y .... , y i<R. Then So bas (s - R)-diut«#sional uteasure satisfging where the implicit constant entries of .4. l's-n(So) >>A l/ïnogradov's notation depends on s and R and the Proof. We apply the lemma of Nadesalingam and Pitman to the R linear forms Ml(y) .... ,]ln(Y) defined by ]l,(y)= Al/nLi(y) for 1 <_ i _< R, in order to relax the requirement A1 _> 1 of their lemnm. We note that there is a slight difference between the definition of [[All that ve use and the definition they use, which accounts for the change in the condition (8.29). Here xve have also ilnplicitly used the last equation on page 704 of [15] to show that the terln H(L) in the lelnnm of Nadesalingain and Pitman is positive. [] Now we return to the proof of Lemma 8.7 and apply Lemma 8.8. By (8.26) and the assumption T _> 1, we mav apply the lelnlna, with the choice Q = (SP )/2. for any w = ('1, t/-'2 ..... wR) with (8.30) lnill (1,/NI/R ) 3T We obtain forl<i<r, and I wil< rein (1" Al/n) for 3 r+l<i<R. p,Æ-n(So(W, .... wh)) » pa(,-n). Now, for anv choice of (Yn+, Yn+_ .... , Ys) 0(iI'l, .-- , wR), there exist real llUm- bers Yl,...,Yn e [-Q, Q] with Li(y) =wi for 1 < i < R. Defining z as iii (3.6), ,ve h ave Li(Pez+y) =wi for 1 <i</ï'. By (3.6) and our choice of Q. we also have P z + y [ P , I + a P ] C [ P P ] SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2711 Recalling the defilfiti(m of SP, T, we sec that we have lts (SP, T) >>F, T-"P Cmbining with (8.27) and (8.28), we see Ihat Ihere is a positive rem constant cl = ci(F, k, R, r, s, 5, z) such that one has (8.31) :T(P) >_ Cl Ps-Rk. This completes the proof of Lelmna 8.7. [] Combining Lemmas 8.6 and 8.7 yields the following lower bound for 2-(P). Lemma 8.9. Suppose that we are in the setti9 of Theorern 1. ad that the coefficient ruatriz A of the systeru F satisfies (3.2) and (3.3), and that there is a real vector z satisfyin9 (3.6). Assume also that we bave IIFII > 1. Define if(P) as in (8.6). Then there is a constant ca = ca(F, h, R, r, s, 6, z) such that for P > ca, ont? bas Z(P) » ps-Rk. Here the iruplicit constater m l ïnogradov's ot«tion may depend on F, k,s. and the special real vector z. but t does hot depend on P. 8.4. Completion of the treatment of the major arcs. We wrap up our work on the major arcs with the following lemma. Lemma 8.10. Suppose that we are in the setting of Theorem 1. and that the coefficient matrix A of the system F satisfies (3.2) and (3.3), and that there s a real v«ctor z satsfymg (3.6). Assume also that we bave IIFI[ _> 1. Then there are constants c4 and c, whzch may depend on F, k, s, R, r, and z, but which do hot depend on P, such that for real numbers P satisfymg P >_ ca, one bas s f. 1-[ -(-/ 1-[ K(,)d. _> cP-. j=l i=r+l Pro@ Choose e = 1/(2k) and apply Lemma 8.2. Since we have g _> 2h + 1 and by the definition (4.10) of B, we obtain .« R (8.32) /.M HgJ(a) H Iç(oi)da-Z(P) « PS-nk(logP)-/(st¢(l+l)) j=l i=r+l Since condition (iv) of Theorem 1.2 holds, one has » 1. By Lemma 8.9. there are constants c3 and c that do hot depend on P such that one has (8.33) Z(P) >_ cP -Rk for P _> c3. Lemma 8.10 follows from (8.32) and (8.33) and t.he bound 1,9 » 1. [] 9. COMPLETION OF THE PROOF OF THEOREM 1.2 In this section, we gather together all of our results in order to complete the proof of Theorem 1.2. We recall that we demonstrated in Section 3 that we may assume that we have e = 1, that we have ]IFII _> 1, that the coefficient matrix A of the system F satisfies (3.2) and (3.3), and that there is a real vector z satisfying (3.6). 2712 EHIC FREEMAN \Ve first observe how one proves the last sentence of Theorem 1.2. namely that and we define m0 (,'.'.) as in (1.4) a,,d assume that we bave if have we 1. HI 0 8.4, noting that we «ertainly have { 2k + 1 for a suciently large «hoi«e of the constant C. For k = 2, we may apply Lemma 8.5. Thus we have >>F 1. Now we turn to the central result of Theorem 1.2. ecall from (4.9) that we have s R (p) ,. g)(o) Iç()da, (9.1) ' J--I i=r+l wllere (P) was dcfincd to bc thc numbcr of solutions of the svstem (4.1) with .r M (P, P) for 1 j s. We first choose a flmcti(m T(P) as in Lcmma 5.5. Xç can now treat the minor arcs and trivial arcs. Bv Lemmas 6.3 and 7.1, one obtains s R X now «onsider the major arcs. Bv Lemma 8.10, we have " j=l i=r+l for P ca, where ca and cs are constants that do hot depend on P. Together with (9.2), it follows for su«iently large P that one has C5 ps--Rk By (9.1), for suciently large P, we have This establishes Theorem 1.2. As a final observation, we note that we llave obtained a lower bound of the expe«ted order of nmgnitude for the number of solutions of out svstem in a box of size P, for ail su«ient.ly large positive P. Recall that we assumed that we bave e = 1, that we bave I[Fll 1, that the coe«ient matrix A of the system F satisfies (3.2) and (3.3), md that tllere is a real vector z satisfying (3.6). Using standard t.e«hlfiques, one can check that under the conditions of eitller Theorem 1.1 or Theorenl 1.2, without mly of these simplifying assumptions, the sanie lower bound holds for suciently large P. X note that in this case, P nmst be suciently large also in erms of e, and the implicit constant in the lower bound for (P) depends Oll . PtEFERENCES 1. M. Aigner, Combiïatorial theor9, Grundlehren der Mathematischen Wissenschaften, Springer- Verlag, New York/Heidelberg/Berlin, 1979. MH 80h:05002 2. H. C. Baker, Diophaïtiïe iïequalities, London Mathematical Society Monographs. New Series. 1. The Clarendon Press. Oxford University Press, New York, 1986. MI 88fi11021 SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2713 3. V. BenIkus and F. G6tze, Lattice point problems and distribution of values of quadratic forms, Ann. of Math. (2) 150 (1(.)!}9), no. 3, 977 1027. MR 2001b:11087 4. B..I. Birch and H. Davcnport, Indejïnite quadratic forms in many variables, Mathematika, 5 (1958), 8-I2. lkII-I 20:31()4 5..I. Br/idern and R. J. Cook, On simultaneous diagonal equations and inequalities Acta Arith. 62 (1992), 125-149. lkIR 93h:11036 6. H. Davenport. Analytic methods for DiopImntine eqtations and Diophantine ineqvalities, Ann Arbor Publishers, Am Arbor, MI, 1963. MR 28:3(102 7. H. Davenport and D. J. Lewis, Simultaneous equations of additive type, Phiios. Trans. Roy. Soc. London Ser. A 264 (1969), 557-595. lklR 39:6848 8. J. Edmonds, Minimum partition of a matroid into independent subsets, .I. Res. Nat. Bureau Standards 69B (1965), 67-72. MR 32:7441 9. D. E. Freeman, A note on one cubic Diophantine inequality, .I. London MaIh. Soc. (2), 61 (2000), no.1, 25 35. MR 2001c:11{43 10. __. Quadratic Diophantine inequalities, .I. Number Theory, 89 (2001). no.2. 268 307. 11. __, Asymptotic lower bounds for Diophantine inequalities, to appear in lkIathcmatika. 12. __, Asymptotic lower bounds and formulas for Diophatine inequalities. o appear in the Proceedings of the Millenuial Conference in Nulnbcr Theory. 13. L. Low..1. PiIman and A. V'olff, Simlianeous diagonal congruences, .I. Number Thcory 29 (1988). 31-59. lkIl 89g:11030 14. I. D. Mcir, Simultaneous diagonal p-adic equations, MathcmaIika 45 (1.(198). 337-3t9. lkI[ 2000k: 11052 15. T. Nadesalingam and .I. Pitman, Bouds for solutions of simultaneous diagonal equations of odd degree, Théorie des nombres (Québec, PQ, 1987). 703-734, de Gruvter, Berlin, 1989. MR 91f:11021 16. __, Simultaneous diagonal inequalities of odd degree, .I. Reine Angew. MaIh.. 394 (198.(}), 118-158. IIR 91c:11019 17. V. M. SchmidI. Diophantine inequalzties for forms of odd degree, Ado. Math. 38 (1980). 128-151. lkIR 82h:10033 18. __, Sinultaneous rational zeros of quadratic forms, Seminar Delange-PisoI-Poitou 1981. Progress in Math., Vol. 22, Birkhiuser, Boston, MA, 1982, 281 307. hIR 84g:10041 19. R. C. Vaughan. On Waring's problem for cubes, J. Reine Angew. MaI h. 365 (1986). 122-170. hIR 87j:11103 20. __, On Waring's problem for sm.aller exponents. H, Mathematika 33 (1986), 6-22. MR 87j:11104 21. __, The Hardy-Littlewood method, 2nd ed., Carnbridge Tracts in Mathematics 125. Cam- bridge University Press, Cambridge. U.K., 1997. MR 98a:11133 22. Y. \Vang, Diophantine equations and inequalities in algebraic number jïelds, Springer-Verlag, Berlin/Heideiberg/New York, 199l. hlR 92a:11036 23. T. D. Vboley, New estimates for smooth tl/eyl sums, J. London Math. Soc. (2) 51 (1995), 1-13. NIR 96e:11109 DEPARTMENT OF IATHEMATICS, UNIVERSIT'. OF ('OLORADO. 395 UCB. BOULDER, ('OLORADO 80309 Current address: School of MaIhemaIics. lnstitute for Advanced SIudv, 1 Einstein Drive. Princeton. NJ 08540 E-mail address: freem@ias.edu TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2715-2732 S 0002-9947(03)03200-8 Article electronically published on Match 19, 2003 ON THE CANONICAL RINGS OF COVERS OF SURFACES OF MINIMAL DEGREE FRANCISCO JAVIER GALLEGO AND BANGERE P. PURNAPRAJNA ABSTRACT. In one of the main resuIts of this paper, we find the degrees of the generators of the canonical ring of a regular algebraic surface X of general type defined over a field of characteristic 0, under the hypothesis that the canonical divisor of X determines a morphism ç from X to a surface of minimal degree }'. As a corollary of our results and results of Ciliberto and Green, we obtain a necessary and suflïicient condition for the canonical ring of X to be generated in degree less than or equal to 2. We construct new examples of surfaces satisfying the hypothesis of our theorem and prove results which show that many a priori plausible examples cannot exist. Our inethods are to exploit the Oy-algebra structure on ç,Ox. These methods have other applications, including those on Calabi-Yau threefolds. We prove new results on homogeneous rings associated to a polarized Calabi-Yau threefold and also prove some existence theorems for Clabi-Yau covers of threefolds of minimal degree. These bave consequences towards constructing new examples of Calabi-Yau threefolds. INTRODUCTION The canonical models of surfaces of general type have attracted the attention of lnanv geolneters. The questions on projective normality and ring generators of the canonical ring are of particular interest. Kodaira [Kod] first proved that [K.ç ' ] embeds a lnilfilnal surface of general type X as a projectivdy norlnal varietv for ail m _> 8. This was later improved by Bombieri [Bo], who proved the saine result if m _> 6, and by Ciliberto [Ci], who lowered the bound to m _> 5. We proved in [GP1] more general results on projective normality and higher syzygies for adjunction bundles for an algebraic surface. As a corollary of these results we recovered and ilnproved the results of Bombieri and Ciliberto on projective normality, and extended them to higher syzygies. We also recovered and extended the results of Reid [RI on the ring generators of the canonical ring of a surface of general type. An important class of minimal surfaces of general type comprises hose whose canonical divisor is base-point-free. Surfaces with base-point-free canonical divisor fall naturally into two categories corresponding to the division of curves of genus greater than one into hyperelliptic and non-hyperelliptic: hose whose canonical Received by the editors July 5, 2002. 2000 Mathematics Sub2ect Classification. Primary 14J29. The first author was partiMly supported by MCT project number BFM2000-0621 and by UCM project number PR52/00-8862. The second author was partially supported by the General Research Fund of the University of Kansas at Lawrence. The tïrst author is grateful for the hospitality of the Department of Mathematics of the University of Kansas at Lawrence. )2003 American Mathematical So¢ietv 2716 F1RANCISCO JAVIE1R GALLEGO AND BANGE1RE P. PURNAP1RA.INA lnorphisln maps onto a sm'fa.ce of lninilnal degree a.nd those whose canonica] mor- phism does hot nlap to such a surface. By a surface of minilnal degree we mean a nondegeuerate surface iu projective space whose degree is equal to its codimension pins 1. The surfaces of minimal degree are classically knowu: they are (linear) P2. the Verouese surface in pa, and smooth rational scrolls or coues over a rational nor- mal curve (see [EH]). Note that, even though we are drawing an analogy between hyperelliptic curves and surfaces of general type whose canonical morphisin inaps outo a surface of miuima.1 degree, the theory is nmch harder for surfaces, because higher degree covers are involved. Examples of such covers are shown in Section 3. In this paper we deal with surfaces of general type whose canonical morphism maps onto a surface of miuimal degree. The surfaces whose canonical morphism does hot mal» onto a surfitce of minimal degree have been studied bv Ciliberto aud Green (sec [('i] and [G]). (',reen and Çiliberto proved the following beautiflfl result regarding the geuerators of the cauouical ring: Let .ç bc a r«gtlar su':face of gcnc'ral t.pc witt basc-point-free canonical divisor. .4ssune that the caonical ltO't]istt ç satisfies the following coditions: (1) ç does ot ,nap X generic«dlg 2 : 1 onto the projecti,e plane: () ç(A-) is ,wt a surface of mi,smal degr«e (other tfian liwar P). Thon thc «am,i«al rig of X is gcnerated in dcgre Icss than or cqual to 2. The smfaces of general type X whose cauonical morphism ç maps X outo a others, where they play a central role in the classification of surfaces of general type with small cî aud in qucstious about degenerations and the moduli of surfaces of general type. The studv of these sm'faces h a direct bearing on the studv of linear series on threefotds such as Calabi-h%u threefolds, as the resnlts in [OP], and the authors" results in [GP2] and Sectiou 4 of this article show. The stmty of the ca.nouical rings of these surfaces is carried out in Section 2. determine the precise degrees of the generators of its canouical ring (see Theorem 2.1). The answer depends on the degree of ç and the degree of Y. As a corollary of out result and the result of Çiliberto and Green (see Corollary 2.8). we fiud that conditious (1) aud (2) above characterize the regular surfaces of geueral type with base-poiut-fi'ee cauonical lmndle whose cauonical ring is generated iu degree less than or equal to 2. This result is surprisiug, because it contrasts with the situation for higher-dimeusional varieties, which, as ve show in [GP3], differs Dom the sit- uatiou for smfaces, ludeed, ve show lu [GP3] that there is uo higher-dimensional aualogue of Corollary 2.5, aud therefore there is no converse of Green's result (cf. [G], Theorem 3.9.3) for higher dimeusioual varieties of geueral type. Iu Section 2 we explain how to use the Oy-algebra structme ou ç.Ox to fiud the multiplicative structure of the canonical ring of X. Even though we reduce the problem Dom a complicated variety to a simpler variety. a surface Y of minimal degree is, there are ceïtain diculties that arise in the process. The proof of Theorem 2.1 involves the study of multiplication of global sections of line bundles on a smface X of general type. To do so we reduce the problem to and this amonnts to stndying maps of muhiplication of global sections of vector bundles instead of line bnndles. This is the first dicnlty. Moreover. the relation between the mnltiplication maps of global sectious of line buudles on X aud the multiplication maps of global sections of vector bundles on is goverued bv the CANONICAL RINGS OF COVERS OF SURFACES OF MINIIvIAL DEG1REE 2717 O--algebra structure on ,Ox. However, we prove our results for a large class of surfaces X. Se their canonical lnorphisln g) niight, and a«tually does, correspond te niany, quite diverse Oy-algebra structures. This is in sharp «ontrast te hyperelliptic «urves, where canoni«al nmri)hisms correspond te degree 2 algebra structures, which are ail quite similar aud very easv te des«ribe. The algebra structures arising frein canonical niorphisms of surfaces are niuch lnore «ompli«ated and liarder te determine. This is the second difficulty one encounters, a diffi«ulty that we are able te overcome in the «ontext of this Imper. In Section 3 we «onstruct new exaniples of surfaces of general type mapping te a surface of niinimal degree and also re«all seine known eues. It is interesting te know in general what positive integers o««ur as degrees of the canonical morphisni if the image is a surface of minimal degree. An answer te this question is helpflfl in finding new exmnples. Having this I)hilosoI)hy in nfind, we prove resuhs which show that seine natural wavs te «onstruct examples de net work. For instance, il, follows ff'oto the results in Section 3 that odd degree «overs of smooth scrolls or «vcli« covers of degree bigger than 3 of surfaces of mininml degree, indu«ed bv the canoni«al morlhisni, de net exist The results on surfaces of general type mentioned above have ramifi«ations for Calabi-Ymt threefolds. In Sectkm 4 of this article we apply these results te obtain new results for a polarized Calabi-Yau threetbld (X, B) with B a base-point-free and ample diviser. Aniong other things, we find out the degrees of the generators of the homogeneous ring asso«iated te B, and we give a chara«terizatiou of polarized Calabi-Yau threefolds (X. B) whose asso«iated holnogeneous riug is generated in degree less than or equal te 2. The construction of examples of Calabi-Yau three- folds has evoked interest in recent vears. One of the iniportant sources of these examples is te take «overs of threefolds of minimal degree. In Section 4. we prove some existence theorems for Calabi-Yau «overs of threefolds of miuimal degree. For instance, frein these results, whi«h are more general, it follows that a Calabi-Yau cover of prime degree greater than 3 induced by a complete linear series canner corne frein a group action. We will expand on these ideas in two forthçoming articles, [GP3] and [OP5]. In the first we study the canoui«al ring of higher-diniensional varieties of general type whose canoni«al morphisln maps ente a va.riety of minilnal degree. One of the results in [GP3] shows that the converse of the theoreln of Ciliberto and Green for surfaces proved in this article is false for higher-dilnensional varieties of general type. In the second we «arry out a detailed study of homogeneous rings associated te line bundles on trigonal «urves. 1. PRELIMINARIES Convention. Tbroughout tbis article we will work over an algebraically closed field of characteristic O. [11 this section we will recall seine known fa«ts about the push-forward of the structure sheaf of a variety by a fiat, finite morphisni. We summarize these fa«ts below, and refer for the proof te [HM], Section 2. Let X and 1" be algebraic varieties. Let 7r : X Y be a finite, fiat morphism of degree n. \Ve have the following facts: 1.1. The sheaf Tf.ex is a rank n, locally flee sheaf ou 1- of algebras over Or. 2718 FRANCISCO 3AVIER GALLEGO AND BANGERE P. PURNAPRAJNA 1.2. There exists a nlap 1 -tr" 7r.Ox Oy of sheaves of Oy-modules defined locally as follows: Given a E rc.Ox, we consider the honolnorphisn of Oy-lnodules .Ox .Ox indu('ed by umltiplication bv . Then we define tr(a) as the trace of such a homomorphisln divided by n. 1.3. £tr is surjective; in fact, thc lnap Oy u.Ox induced bv u is a section of 2tr. Therefore the sequence t 0E.Ox O 0 split.s. E is the kernel of %tr and locallv consists of the trace 0 elements of will call E the trace-zero module of u. 1.4. u.(Ox) is a sheaf of Oy-algebras; therefore, it h a multiplicative structure. It.s multiplication nmp is an O -linear map [O-eE][O-e] O,-eE ruade of four components. The first component is given bv the multiplication in Oy, and therefore goes to Oy. The components OrNE O)-oE, E NO)- are given by the le[t and right module structure of E over Oy. and therefore go to £. Finally, there is a fourth component which factors through S2 E 0- E , for nmltiplication in .Oy is conmmtative. 2. COVERS OF SURFACES OF MINIMAL DEGREE Out purpose in this section is to study the generators of the canolfical ring of certain surfaces of general type. Specifically, we are interested in studying those regular surfaces of general t.vpe whose canouical divisor is base-point-free and such that the ilnage of the canonical morphism is a variety of minimal degree. We obtain the following result. Theorem 2.1. Let S be a regular surface of general type witl af worst caonical singularities and uch tlat its canonical bundle Ks is base-point-free. Let ç be the canonical morphism of S. Let n be the degree of ç and assume that tle image of is a surface of minimal degree r. Then: 1) if n = 2 and r = 1 (i.e., if is generically 2 : 1 onto p2), then the canonical ring of S is generated by its part of degree 1 and one generator in degree 4: 2) if n 2 or r 1, then the canonical ring of S is generated by its part of degree 1. r(n - 2) generators in degree 2 and r - 1 generators m degree 3. CANONICAL IRINGS OF COVERS OF SURFACES OF MINIMAL DEGREE 2719 The knowledge of how manv linearly independent generators are needed in each degree is obtained from the knowledge of the image of the multiplication maps of global sections of powers of the canolfical bundle. We study those multiplication lnaps by studying similar lnaps of a curve C in IKs]. Thus we will first prove the following proposition. Proposition 2.2. Let C be a smooth, irreducible curve. Let 0 be a base-poi77t-free line buTTdle o C such that 0 °2 = Kv. Let re be tbe rnorpbism induced bg I01, let be the degree of rr. and assume that re(C) is a ratioal normal curve of degree r. Let ff(s, t) be the multiplication map H°(O ®) ® H°(O 'st) H°(O'r-+t), for all s, t > 0. The codimension of the image of /3(s, t) i H°(O ®s+t) is as follows: a) If r = 1. the codimension is: a.1) n- 2. for s= t = l, a.2) 0, for s = 2, t = 1. i.e., 13(2, 1) surjects, a.3) 1, fors=3, t= 1, a.4) 1, fors=t=2, n = 2 and O if n >2. a.5) 0, for s >_ 4, t = 1, i.e., p(s, 1) surjects for all s >_ 4. b) If r > 1. the codirnensio is: b.1) r(n - 2). for s = t = 1, b.2) r-l. fors=2, t=l, b.3) 0. for s > 3, t = 1. i.e., ff(s, 1) surj«cts for all s > 3. Moreover, if r= 1 and n = 2. then the image of/3(2, 2) and the image of/3(3, 1) are equal. In order to prove Proposition 2.2, we will use the following. Lemnla 2.3. Let C. 0 a'nd re be as in the staterneTt of Propositio 2.2. Then rc.Oc = Op, ( (n -- 2)O1-, (-r -- 1) ( Op, (-2r -- 2) . Pro@ Since the ilnage of rr is Slnooth and of dilnension 1, rr is fiat. Then îr.Oc = Op @ E as Op,-lnodules, with E a vector bundle over P of tank n - 1. We noxv show that e = (n- 2)Ol:,,(-r- )¢, Ol:,,(-2r- 2). We have rr.0 = rr.Oc x Op,(r) and rr.tçc = rc.Oc ® Op,(2r), bv the projection formula. Any vector buudle over P splits: hence g.O C = Op (D E = Op (90p (al) -'- @ (._9pa (an-l) for solne negative integers al,..., an-1 (C is connected). Then hi(Içc) = 1 ilnplies that exactly one of the ai's, let us say an-i, satisfies an-1 ÷ 2r = -2. On the other hand, since re is induced by the complete linear series ]0[, h°(O) = r + 1 = h°(Op (r)); so ai + r _< -1 for ail 1 < i < n - 2. Filmlly, since the degree of 0 is g(C)-l, we have bi(0) = h°(O) = r+l. Sillce hl((,Qp (-r-2)) --- r+l, ai+r >_ -1 for ail 1 < i < n - 2, and so ai ÷ r = --1 for ail 1 < i < n - 2. [] (2.4) Proof of Proposition 2.2. In Lellmm 2.3, we have COlnpletely deternfined the structure of r.Oc as an Opt-lnodule. Now we look at the structure of rr, Oc 2720 FRANCISCO .lA\ IER GALLEGO AND BANGERE P. PURNAPRA,JNA as an Op,-algebra. If ,t = 2. it is completely detelmined by the t)ranch diviser of 7r on P. since in this case rr is cyclic. If 7t > 2, we observe the following: For seine 1 < i.j < 7 - 2. the projection of the map (2.4.1) Op1 (ai) @ Op, ((tj) 7r.O C te Op, (-2r - 2) is surjective: in fact. it is an isolnorphism. This is se because otherwise Op1 .) Op1 ((/1) t "'" Ç' Op1 (Ort--2) would be an integra.1 subalgebra of rc.Oc, free over Op of rank , - 1. Then n - 1 should divide 7. which is net possible if 7 > 2. New we will use ont knowledge of Tf.ex- te stndv the lnaps/3(s, r) which appear in the statelnent of the proposition. We will write /4 iii place of /3(s. 1). Let R = H(O). Then, since 0 = Tf*Op, (r). bv the projection formula. RI = H0(OpI (r)). nl ---- H°(Op, (lr)) (, - 2)H°(Op, ((1 - l)r - 1)) ( H°(Op, ((l - 2)r - 2)), /1+1 = H°(OP l (Il n u 1)r)) @ (,, - 2)H°(Op, (lr - I)) @ H°(Opl ((l -- 1)r - 2)). Therefore an element of Rt. i.e., a global section of H°(O"l), is a sure of n com- I)onents, one in each piece of the al)ove decoml»osition of Rt. On the other hand, the product of an elelnent of Rt belongilg te one of the blocks with an elelnent of R is determined bv the ring structure of (gp and bv the module structure of More precisely, the lestriction of/3t te H°(Op (If)) ® H°(Op (r)) lnaps, in fact isolnorphically, ente tt°(Op ((! + 1)r)). The restriction of t3t te each of the blocks H°(Op ((l - 1)r - 1)) ® H°(Op (r)) naps te the corresponding H°(Op, (/r - 1)). This restriction is 0 if (l- 1)r- 1 is negative and an isomorphism otherwise. Likewise, the restrict.ion of pt te H°(Op((1 - 2)7" - 2)) ® H0(Op1 (r)) goes te H°(Op((! - 1)r - 2)), being 0 if (1 - 2)r- 2 is negative and an isomorphism otherwise. Theretbre it is crucial te tell which blocks of a given R are 0. \Ve have I = H°(OP '(r)), R2 = H°(Opl(2r)) ( (,, - 2)H°(Op1 (7"- l)). and if 1 > 3. nl = H°(Op,(lr)) (Tt - 2)H°(Op, ((l - 1)r - l)) H°(Op, ((I - 2)r - 2)) . Ail the direct smmnands appearing in the above formulae are nonzero, except H°(C)p,((1- 2)r- 2)) when l = 3 and r = 1 and (n- 2)H°(Op,((l- l)r- 1)) for all ! and all r when 7 = 2. We new deterlnine the image of fit- If 1 = 1. the image of/31 is H°((._gp (2r)), which has codimension (7 - 2)r in R.2. If 1 = 2. the image of/32 is H°(Op, (3,')) @ (7 - 2)H°(Op, (2r - 1)), which bas codimension r - 1 in Ha. If 1 = 3 and r _> 2 or if 1 _> 4. the image of surjects. All this proves a.1). a.2), a.5) and b). If r = 1. the image of/3(3.1) is H°(Op,(4r)) (n - 2)H°(Op(3r - 1)). which has codimension 1 in /74. This proves a.3). If r = 1 and n = 2. the image of/3(2.2) is H°((._gpl (4r)). which has codimension 1 irt /4. This proves the first claire iii a.4) alld the last sentence of Proposition 2.2. Finally, if n > 2. recall (sec 2.4.1) that for seine 1 _< i.j <_ 7 - 2. the projection of the lnap (._p1 (I/i) ("., (._p1 (aj) Tf.(._ C CANONICAL FIINGS OF COVEIClS OF SUIClFACES OF MINIMAL DEGIClEE 2721 to Op1 (--4) is sm:jective; in fact, it is an isolnorphism. Then. if 7 > 2, the image of ff(2, 2) is all R4. This proves the second part of a.4). Remark 2.5. Note that 0 = Içc- Then a proof of a.4), alternate to the one given above, can be obtained ffoto Noether's theorem and from the base-point-free pencil trick. The way in which Noether's theorem is related to the algebra structure of w.Oc is shown in [GP4], where we will give a diflrent., simple proof of this classical result for a general curve in M. From Proposition 2.2 we obtain the following. Corollary 2.6. Let C be a smooth curve. Let 0 be a base-point-free line bundle o C such that 0" ' = Kc. Let be the mohi,sm iduced bg 0], let be the degree of and assume that n(C) is a ratioml normal curve of degree r. Let R be @o H°(o';)" Ten: 1) if r = 1 ad = 2. lhc rig R is g«,cratcd byit.s part qf degree 1 and one generator in degree 4: 2) if r = 1 atd o > 2. te 'itg R is g«n«rated bg ifs part of degree 1 gcterators in d«g've 2: 3) if r > 1. the ring R is gcerated by its part of degree 1. r( - 2) generators i?t degree 2 and r- 1 gcn«rators it degree 3. Pro@ To know in what dcgrees we need generators, we look at the maI)s ff(s. t) of multiplication of sections. Preciselv the mmfl)cr of generators needcd in degrce 1 + 1 is th(' codimensiou in R+ of the smn of th(' images of /3(1,1). - ., , [W]). In particular. is gcnerated in degrce less than or equal to 1 if snrjects for all k 2 l. Thus 1) folloxvs from part a) of Proposition 2.2 and ff'oto the fact that the images of (3.1) and (2, 2) are equal. 2) follows likewise ff'oto part a) of Proposition 2.2 (note that in this case ff(2.2) surjects). Finally, 3) follows ff'oto part b) of Proposition. (2.ï) Proof of Theorem 2.1. The proof rests on Proposition 2.2. The idea is "to lift" the generators of R to the canonical ring of S. Let us define 0 -Ns 0 (i(s) (ç2) u((s.-,+,, - and let us denote a(s. 1) as a. As in the case of R, the images of a(s, t) will tell us the gencrators of each graded picce of the canonical ring of S. In fact, it will suce to prove the following: (a) If r = 1 and = 2. a surjects for all 1 1, exccpt if 1 = 3. The images of aa = a(3, 1) and a(2.2) are equal and have codimension 1 in H°(K4). (b) If r = 1 and n > 2, a surjects for all 1 1, except if 1 = 1,3. The image of al bas codimension n - 2 in H°(K). The map a(2.2) is sm'jective. (c) If r 2, a is surjective if 1 OE 3. The image of a has codimension r( - 2) in H°(Bç). The image of aa has codimension r 1 in H°( - Thus we proceed to prove (a), (b), (c). Recall that I = ç(S) is an irreducible variety of lnilfimal degree and, in particular, normal. On the other hand, the locus of the points of I" with non-finite fibers bas codimension 2. Thus, using Bertini's theorem, we can choose a smooth curve C of IKs[ such that the restriction of the canonical morphism of S to C is finite (and fiat) onto a smooth rational normal curve of degree r. Let us denote by 0 the restriction of I@ to C Bv adjunction, 2722 FRANCISCO JAVIER GALLEGO AND BANGERE P PURNAPRAJNA Kc' = 0 @z. Since fçs is base-point-ffee, so is 0. Finally, since HI(O\ is induced by the complete linear series [0 I, and therefore C, 0 and satisfy the hypothesis of Proposition 2.2. ë prove first thc statements in (a), (b) and (c) regarding the maps t- Consider the following conmmtative diagram: o( -t HO(h-s) Ho(K) Ho(o) H°(K t) H°(Os) .... s o -e ,,o ( ,-et+ x H o (Oat+x) H (K s ) ç > 's The rightmost horizontal arrows are smective because Ht(Os) = 0. bv Serre duality and by Kawamata-Viehweg vanishing. The left vertical arrow triviaIIy sur- jects. The right vertical arrow is the composition of the map H°(A H°(O t) H°(O), which is surjective for all l I again because H(Os) = O. by Serre duality and by Kawamata-Viehweg vanishing, and the nmp t of multiplica- tion of global sections on C, studied in Proposition 2.2. Then if foIlows from ching °t, t+ H°(O @+) nmps the image of t onto the diagram tlmt the mal) ,, "s the image of t, and that the codimension of the image of t in H°(O t+) is equal to the codimension of the image of t in H "s » This, together with Prop sition 2.2. a.I.a.2, a.3, a.5 and b. proves the claires in (a), (b) and (c) concerning the codimensions of the images of the maps t. Thus the only things left fo prove are the claires al)out (2.2) when consider the commuttive diagram o - Ho(z) HO(O.») H°(Iç') H°(Ks) H°(Iç2) H (h s ) H°(Iç 3) ç > H°(Iç 4) » H°(O ) The rightmost horizontal arrows are surjective because H (Os) = 0 and bv Serre duality, and by Içawamata-Viehweg vanishing. The left vertical arrow surjects, we bave ah'eady proven. The right vertical arrow is the composition of the map H 0 -2 0 (Ks)H (0 ) H°(O)H°(O2), which is surjective because S is regular and by Serre duality, and the map (2, 2) of multiplication of global sections on C. Then if follows from chasing the diagram that the map 0 H (I; s ) H°(O ) maps the inmge of (2.2) onto the image of ç(2.2), and that the codimension of the image of (2.2) in H°(O ) is equal fo the codimension of the image of (2, 2) in 0 H (K s ). On the other hand, we know that the image of (2.2) and the image of ç3 ç(3. i) are equal of codimension 1 in H°(Oa). if r = 1 and = 2. Thus we conclude that the inmges of (3, i) and (2, 2) in H°(Iç ) are also equal and of codimension i. Finally, if r = 1 and n > 2, ç(2.2) surjects by Proposition 2.2.aA. Thus we conclude that if r = 1 and n > 2, then (2, 2) surjects. Theorem 2.1 compIements known results on generation of the canonical ring of smooth, regular surfaces of general type. Ciliberto and Green (cf. [G], Theorem 3.9.3, and ICi]) proved that, given a smooth surface of general type with h.(Os) = 0 and h-s globally generated and ç being the canonical morphism, a sucient condition for the canonical ring of S fo be generated in degree less than or equal fo 2 is that: CANONICAL IRINGS OF COVERS OF SURFACES OF MINIMAL DEGREE 2723 (1) does hot map S generically 2 : 10lltO p2, and (2) ç(S) is hot a surface of minimal degree other than linear p2. As a corollary of the Ciliberto and Green result and of Theoreln 2.1, we obtain the following: Corollary 2.8. Let S be a smooth regular surface of general type and such that Ks is globally generated. Let ç be the canonical morphism of S. The canonical ring of S is 9enerated in degree less than or equal fo 2 if and only if (1) ç does hot map 5; 9enerically 2 : 1 onto p2, and (2) ç(S) is hot a surface of minimal degree other than linear p'2. 3. EXAMPLES OF SURFACES OF GENERAL TYPE bi this section we construct some lleW exanlples of surfaces of general type that satisfy the hypothesis of Theorem 2.1. The easiest way one could think of producing examples would be to build suitable cyclic covers of sm'faces of mininlal degree. However, as the next proposition shows, onlv low degree cyclic covers can be induced bv the camnlical morphisnl of a regular surface. So we have to employ other means to construct these exanlples. Proposition 3.1. Let X be a surface of general type with af worst canonical singu- larities and with base-point-free canonical bundle. Assume that the complete canon- ical series of X restricts to a complete linear series on a general member of the canonical series (e.g., if X is regular). Let ç : X be the canonical morphi.«m fo a surface of minimal degree. Let n be the degree of ç. Let I, be a smooth open set of Y whose complement bas codimension 2 and let L be a line bundle on U. Assume that (ç.Ox)lu = Ou @ L - @... @ L'='-' Then n = 2 or 3. Pro@ Let H be a gelleral hyperplane section of }" contained in U and let C be the inverse image of H by ç. Then C is a slnooth irreducible member of [tç.\-[ and H is a smooth rational normal curve. By assunlption the morphism 91c : C H is induced by the complete linear series of a line bundle 0. Bv adjunction 0 a2 = tçc. Thus C, 0 and çlc satisfy the hypothesis of Lelmna 2.3, and ((,lE).O'-, z Opl e (n -- 2)Opl (--r -- 1 ) @ Opl (--2'" -- 2) . On the other hand, (q)lc).Oc is equal to the restriction of q).Ox to H, and hence (Ic).OC : Op I L t-1 ( . . . ( L ri-n, where U is the restriction of L to H. The onlv way iii which (ç[c).Oc can have these two splittings is when n = 2 or 3. [] Corollary 3.2. Let X be a regular surface of general type with at worst canonical singularities and with base-point-ffee canonical bundle. Let be the image of X by its canonical morphism X Y. If Y is a surface of minimal degree and c2 is a cyclic cover, then the degree of op is 2 or 3. The next proposition also rules out manv possible examples of covers of odd degree: 2724 FRANCISCO JAVIER GALLEGO AND BANGERE P. PURNAPRAJNA Proposition 3.3. Let X be a surfl«e of general type with at u,orst «anoni«al sin- gdarities whose «anoni«al divisor is base-point-free. Let ç be a morphism induced by a su.bseries of ]Kxl. I]' is generically finite onto a smooth s«roll } C Pv. then the degree of is even. In particulor, there are hot generically finite covers of odd degree of smooth rational no,wal scrolls induced bg subseries of Içx. Proof. Let f bc a fiber of I and lct C be a scctiou of Y. Lct -d = C 2. Since Y is a scroll, its hyperplane section is linearly equivalent to C+ tuf, for some integer m. Then ç.x- = *(c + tuf). Then dcg = (*f). (*C) = (*f). (Içx - mç*f) = (ç*f) (Kx- + ç*f), which is an cven number. Now we constru«t some examples of regulm" minimal surfaces X whose canonical uiorlhism maps onto a varietv of minimal degree, and also mention known ones relevant t« this Imper. The cases when is a generically finite morphisui and bas degree 2 or 3 have been completely studicd bv Horikawa and Konno (see [H1], Theorem 1.6, [H2], Theorem 2.3.I, [H3], Theorem 4.1 and [Kou], Lemma 2.2 and Theorem 2.3: see also Mendes Lopcs and Pm'dini, [IP]). As it t«lrns out, there exist generically double covers of linear p2, the Vç, ronese surface, smo,th rational normal scrolls S(a. b) with b 4. and cones over ratioual normal çurves of dcgree 2, 3 and 4 and generically triple covers of pe (in particular, cyclic triple covers of p2 ranfified along a sextic with suitable singularities) and of the cones over rational normal curves of degree 2 and 3. Horikawa (see [H4], Theorem 2.1) also dcscribes all generically finite quadruple covers X ', wherc X is a smooth, nfinimal regular sm'face, ç is the canonical morphism of X, and ) is linear p2. The examples of Horikawa and Komlo just reviewed are examples of covers of degrce lcss than or equal to 3 of surfaces of nfinimal degree and quadruple covers of P. XX now construct three new sers of exmnples of regular surfaces of general type that are quadruple covers of surfaces of minimal degree under the canonical mor- phism. These examples are 4:1 covers of smooth rational normal scrolls isomorphic to the Hirzebruch surfaces F0 and F, and of quadric cones in p3. Example 3.4. IIe constct finite quadruple covers X -, where X is a s,nooth minimal regular surface of general type, is the canonical morphism of X. and )" is a smooth rational scroll S(m, m). m 1. Let f be a fiber of one of the fibrations of P and let ff be a fiber of the other fil»ration. Then " is PI x PI, and it is embedded in p-+l by f + mf'[ or by bi and b2 satisfv the following: and b2 = 1. = nt + 1 . If I" is embedded by If'+ tuf I, let a, a2, bi and b2 satisfy the following: eitherb =l, b2=2. a=m+l anda2=l. or b = 2. b2 = 1, ai = 1 and a2 = m + 1 . For i = 1, 2, let Di be a smooth divisor linearly equivalent to 2(aif ÷ biff) such that D and D2 intersect at D D2 distinct points. Those divisors exist because bv the choices of a. a2, b and b2, both 2(af + blff) and 2(af + b2f') are verv ample. Let X be the double cover of " ranfified along D. Since D is CAN(ï)NICAL 1RINGS OF COVE1RS OF SU1RFACES OF MINIMAL DEG1REE 2725 Slllooth, se is _\-r. Let D. I)e the im-erse image in 3. "r of D2 1)v tgl. Since D 2 is snlooth and lneets D 1 ai distinct i)oints, D is also snlooth. Let X X r be thc double cover of .\' ralnified along D.. Since X and D are 1)oth Slnooth, sois .\. Let ç = ¢'1 o .92. Now we will show that .\" is a regular surface of genel'al type, that Kx = (*O)-(1), and that ç is induced 1).v thc COlnl)lete canonical stries of X. First we find out thc structure of y).(,.9.\- as a module over Or. Recall that ç2,Ox = (gx, @ l*(gr(-a2f - bf'). Tllen @.Ox - I. Ox' t l. (l* OY(--o2f -- b2ff)). Since pl.O\', = Oy @ Ot-(-alf - bf'), then by the l)rojection fornmla we have ç.Ox = Or ©O-(-alf - blf')©Ov(-o2f -b2f')©O-(-(al +a2)f - (bt +b.2)f') . We sec now that X is regular. Rccall that Hl(Ox) = Hl(ç.Ox). Oto" «boite of al, 02, bi and b2 ilnplies that alf + blf' and a2.f + b_f' are both very ample; thus, by Kodaira vanishing, Hl(Oy(-alf - hlf')) : Hl(Oy(-a2f - h2f')) = H'(Or(-(a, + a2)f- (b, + b2)f')) = O. ThelL since H (Or) also vanishes, se de H 1 (ç.Ox) al,(1 H (C9.\-). \\i, nov COml)ute Kx. Since ç2 is a double cover ralnifi(d al(mg D., For a similar reason, I'\- = (2*(Içx' ® ç(O-(a_f + b2f')). IÇx, = yg(/Çy ( Oy(olf -t- blf')). Then Iç\- = ç*(tç-® O}-((o 1 -1- a2)f + (bi + b2)f')). Since/£v = Or(-2f - 2f'), it follows again froln the choi«es of al, 02, bi and b2 that Kx = ç*Or(1). Finally, to sec that ç is indu«ed bv the «olnplete canoni«al linear series of X. we compute H°(Içx). We do the COlllI)lltatiOll iii the case Or(l) = Or(f + tuf'). The case OF(l) = Or(tuf + f') is analogons. Sin«e /\- = * or ( 1 ), H°(tçx) = H°(O-(1)) @ H°(O-((1 -al)f + (m- bi)ff)) @H°(Or((1 - a2)f + (m - b2)f')) Ç-H°(O'((1 - al - o2)f + (m - bi - b2)f')) Again, by the «hoi«es of al. a2, bi and b2, the last three direct bllll18 of the above expression are 0. Se ç is indeed indu«ed by the complete canonical series of X. Example 3.5. H'e construct finite quadruple covers X Y, u,here X is a smooth regular surface of general type with base-point-fcee canonical bundle. ç is the canon- ical morphism of X. and l is a smooth rational scroll S(m - 1, m), m >_ 2. Let C0 be the lnininml section of F1 and let. f be one of the fibers. Then l" is F1, and it. is embedded in p2, by [Ç0 + mfl- Let al, a2, bi and b2 satisfy the following: eitheral =l,a2=2, bl=m+l andb2=2, or al=2, o2=l,bl =2andb2=m+l . For i = 1, 2, let Di |)e a slnooth diviser linearly equivalent te 2(aiCo + bi f), su«h that D1 and D_ interse«t at D1-D2 distinct points. The fa«t that such divisors exist 2726 FRANCISCO JAVIER GALLEGO AND BANGERE P. PURNAPRAJNA follows from our choice of al, o2, b and b2, which irai)lies that of the linear systems of D and D2. one is very ample, and the other is base-point-free. Let X' " be the double cover of }" ramified along D. Since D is smooth, sois X'. Let D be the inverse image in X' of D2 by ç. Since D2 is smooth and meets D transversally, D is also smooth. Let X X' be the double cover of X' ramified alongD. SinceX' andD arebothsmooth, soisX. Let ç=çoç2. Nowwe will show that X is a regular surface of general type, that L'x- = ç*Oy(1), and that ç is induced bv thc complete canonical series of X. First we find the structure of ç.O.x as a module over Or. Recall that ç2.0.x = O.x-, ç*O-(-a2Co - b2f). Then Ç. OX = I.OX' çl.(l*Oy(-a2Co - b2f)) Since ç.Ox, = Oy ç Oy(-aCo - bf), then bv the I)rojection formula we have ç.Ox = O- @ Or(-aCo - bf) Oy(-aCo - b2f) Or(-(a + a)Co - (b + b2)f) Xê sec now that X is regular. Recall that H(Ox) = Hi(ç.Ox). Our choices of al, a2, bi and b2 inll)ly that a.( + bf and a2Ço + b2f are both be-point-free and big divisors: thus. by Kawanlata-Viehweg vanishiug, H(Oy(-aaCo - bf)) = H(Oy(-a2Co - b2f)) = H'(Oy(-(a + a2)Co - (b, + b2)f)) = 0 . Then, since H(Oy) also vanishes, so does H(ç.Ox -) and therefore H(Ox-). X now COHlpUte KX-. Since ç2 is a double cover ramified along D, tçx = 2*(tçx' ® ç(O)-(O2Co + b2f)). For a similar reason, K\-, = 2(tç ® Or(a,Co + b,f)). Thell Kx = o*(tQ-¢ Ov((a, + a2)Co + (bi + b2)f)). Since tçy = Or(-2Co - 3f), it follows from our choice of a, a2, b and b2 that Kx = ç*Oy(1). Finally, t.o see that ç is induced by the complete canonical linear series of X, we compute H°(Kx). Since H°(tQ) = H°(O(1)) ® H°(Oy((1 - al)Co + (m- bi)f)) (SH°(O((1 - a2)Co + (m - b2)f)) (SH°(OY((1 - al - a2)Co + (m - b - b2)f)) . Again, bv the choices of a, a2, b] and b2, the last three direct sums of the above expression are 0. So Remark 3.6. With the saine arguments, if one allows certain nfild singularities in D1 and D, then one tan construct examples of covers of Fo and F with at worst canonical singularities. Finally, we construct an exmnple of a quadruple cover of a siugular surface of minimal degree. CANONICAL RINGS OF COVERS OF SURFACES OF MINIMAL DEGREE 2727 Example 3.7. 14"e construct an ca'ample of a mnooth, generically finite, quadruple cover X Z of the quadric cone Z in p3, where X is a regular surface of general type whose catonical divisor is base-poit-free, ad y) is its canonical rnorphism. Let 1- = F2. Let C0 be the lninilnal se«tion of ) and let f 1)e a fiber of Y. Let D1 be a Slnooth divisor on 1, linearly equiva.lent to 2C0 + 6f and lneeting C transversally. Let D2 be a Slnooth divisor on 1", lineaïly equivalent to 3C0 + 6f and meeting D1 transversally. Su«h divisors D1 an(i D2 exist, be«ause 2C0 + 6f is very alnple and 3C0 + 6f is base-point-free. Note also that, silwe (3('o + 6f) C0 = 0. C0 and D2 do hot meet. Let X' be the double «over of Y along D1. Sin«e D is Slnooth, so is X'. Sin«e D1 lueets (7'o at two distiu«t points, the pullba«k C of C0 by 991 is Slnooth line with self-interse«tiol -4. Let D bc the pullba«k of D2 by 91. Since D1 and D2 meet tl'ansversally, D. is smooth, and sin«e D2 and C0 do hot meet, neither do D and C D. Let L be the pullba«k of 2C0 + 3f by 9. Let X X' be thc double «over of X' ahmg D U C. Sin«e D U C is Sllooth, so is X. Let 9 = 991 o 992. Thon (3.7.1) ç.O\- = ç.ç2.0x = ç.(O\-, @ L,*) = CO- e, (.gv(-Co - 3f) @ Ov(-2Co - 3f) @ Ov(-3Co - 6f) . Sin«e Co + 3f and 3Co + 6f are big and base-point-flee, bv Kawanmta-Viehweg valfishing and Serre duality, Hl(Ov(-Co -3f)) = H(Cgv(-3Co -6f)) = 0. Bv Serre duality, H (C9v(-2C¢-3f)) = H 1 (C9v(-f))* = 0. Then. since H 1 (C9v) = 0. X is regular. Arguing as in Exalnples 3.4 and 3.5. we sec that (3.7.2) tçx = ç*(tç ® Ov(3Co + 6f)) = ç*Ov(Co + 2f) . Now we compute H°(Kx). Using the proje«tion fbrmula and (3.7.1) and (3.7.2), we obtain that H°(Içx) = H°(Ov(Co + 2/)) @ H°(Ov(-f)) @ H°(Ov(-Co - f)) (H°(Oy(-2Co- 4/)) = H°(Oy(Co + 2/)) . Thus the «anonical morphism of X is the composition of ç and the lnorphisln Y ¢- Z C pa, induced by the «omplete linear selies of C0 + 2f. Sin«e ¢ contra«ts C'o, the canonical norphism of X is hOt finite, but it is generically finite of degree 4 onto Z, which is a surface of nlinilnal degree, as we wanted. On the other hand, if C' is the pullback of C0 by ç, then C' is a smooth line with self-interse«tion -2. Thus the lnorphisln ¢ o ç also fa«tors as ç' o ¢, where x LX is the lnorphisn from X to its canolfical lnodel X and is the canolfical morphisln of X. Thus ç' is an exalnple of a filfite, 4 1 canonical morphisln ff'oto a regular surface of general type with canonical singularities onto a singular surface of lnininml degree. 2728 FRAN(2'ISCO JAVIEF CALLEGO AND BANGERE P. PURNAPRAJNA 4. APPLICATIONS TO CALABI-YAU THREEFOLDS The results proved in Sectons 9 and 3 bave ramifications for Calabi-Yau three- fi»lds. F/ecall that if X is a Calabi-Yau threefold and B is a big and base-point-free divisor, a general member of ]BI is a surface of general type. Then the geometry and properties of surfaces of general type m'e directly related to those of Calabi-Yau threefolds. Concretely, the results we have obtained in Section 2 on the canonical ring of surfaces of general t,vpe can be 'lifted'" to achieve analogous results for Calabi-Yau threefohls in a way similar to the way in which our study of rings of curves Mlowed us to obtain results for surfaces of general type. On the other hand. constructing examples of ('alabi-Yau threefolds has attracted the attention of gcometers in recent vears. One of the important sources for these exmnples is 1)recisely to take covers of varieties of minimal degree. Proposition 3.1. Corollary 3.2 and Proposition 3.3 tell us features of generically finite covers of surfaces of minimal degree iuduced by the canonical morphism. We will see how these features pass on to generically finite morphisms ff'oto Calabi-Yau threefolds to threefolds of minimal degree, and, as a consequence, we will obtain, among other things, that many a priori possible examples of Calabi-Yau threefolds cmmot exist. We start with thc Calabi-Yau threefold aualog of Theorem 2.1: Theorell 4.1. Let X be a Calabi-I'ut ttreefold with al worst canonical sitt9ulari- ries. and let B be a big atd base-po#t-free lie buttdle on X. Let ç be the morphism ind'uced by IBI. Let be the de9ree of ç. attd assnne tha.t the image of ç is a variet9 of mitimal de9ree r. Then " 1) If = 2 atd r = 1 (i.e., if ç is 9enericallg 2 1 oto p3), the caonical rit 9 of X is 9enerated b9 its part of de9ree 1 and on, e 9eerator i de9ree 4. 2) If 7 2 or r 7 1. the cartoical rit9 of X is 9eerated bg ifs part of de9ree 1. r(n - 2) 9erterators in de9ree 2 attd r - 1 9enerators i, de9ree 3. Sketch of pro@ The proof follows the saine lines as the proof of Theorem 2.1. Let us define H°(13 ?) ® H¢'(B '-'-«) "(.____) Ho(B..:+t) . and denote 7(s, 1) as "fs- The images of "r(s, t) will tell us the generators of each graded piece of the ring tï__ 0H°(Bç»). In fact. it would suffice to prove the following: (a) If r = 1 aud , = 2, ",/ surjects for all l >_ 1, except if ! = 3. The images of "fa = 7(3, 1) and 7(2, 2) are equal and have codimeusiou 1 in H°(B-4). (b) If r = 1 and n > 2.7 surjects for all / >_ 1, except if / = 1.3. The image of ? has codimension - 2 iu H°(B»2). The map 7(2, 2) is surjective. (c) If r _> 2, 7t is surjective if 1 _> 3. The image of 'i bas codimension r(t - 2) iu H°(B®). The image of',/ has codimension r - 1 in H°(Ba). Now a suit.able h.vperplane sectiou of the image of ç is an irreducible surface t" of lninimal degree. It.s pullback by ç to X is a surface S of general type with at worst canonical singularities. Moreover, bv adjunctiou B 0 Os = Ks, and the complete linear series of B rest.ricts to the complete canonical linear series of S: so ols is the canonical morphism of S. Therefore S is uuder the hypothesis of Theorem 2.1. and the proof follows verbatim the steps given in 2.7, the role of S there being played by X here, the role of C there being played bv S here aud the role of Proposition 2.2 there being played by Theorem 2.1 here. CANONICAL IINGS OF COVEIS OF SUIFACES OF MINIMAL DEGIEE 2729 To how wha,t we mean, we outline how to find the ilnages of the maps t. consider the following commutative diagrmn: H°(B '») @ H°(Ox) H°(B 5 ') @ H°(B) H°(B :: .) H°(Içs) H°(B TM) ç H°(B ''+1) H The rightmost horizontal arrows are smjective because H (Ox) = 0, by Serre duality and by Kawmnata-Viehweg vanishing. The left vertical arrow triviallv sur- jects. The right vertical m'row is the composition of the map H°(B 0 H (Iç) @ H°(Ns), which is smjective for all I 1 again because H(Ox) = by Serre dualitv and bv Kawamata-Viellweg vanishing, and the map at of multi- plication of global sections on S, stndied in (2.7). Then it follows from chasing the diagram that the map H°(B c+l) n çl S ) lnaps the image of ?l onto the image of al and that thc codimcnsion of the image of or in H°(N/+1) (which was equal to the codimcnsion of the image of t in H°(Ol+l)) is cqual to the codi- mension of 7 in H°(B:»l+l). This, together with the claires in (a), (b) and (c) coneerning the codilnensions of thc ilnages of the maps at proved in (2.7), givcs the codilnension of the ilnages of the lnaps 7l in H°(B l+). The clailns rcgarding 3(2.2) are proved analogously. As we did in the case of the canoniçal ring of regular surfaces of gencral type, we can characterize when @0 H°(B') is gcnerated in degrce less than or equal to 2 using Theorcm 4.1 and results fronl [GP-]" Corollary 4.2. Let X be a Calabi- l hu thre@ld wilh at worst canouical singulari- ries. Let B be a bi9 a,d base-point-free line budle on X atd let ç be the induced by IBI. Thc @o H°(Bl) is generated in degree less than or equal to 2 if and only if (1) ç does hot map X 9eterically 2 1 onto P:" and (2) ç(X) is hot a thre@ld of minimal degrce other than p3. Pro@ Theorem 4.1 tells us that if ç(X) is a variety of nfinimal degree, then the ring @lO H°(Bet) is generated in degree less than or equal to 2 if and only if (X) = p3 and the degree of ç is greater than 2. If ç(X) is hot a varietv of minimal degree, then in the prooN of [GP_], Theorems 1.4 and 1.7. it is shown that HÙ(B .:.' ) @ H°(B 1) H°(B l+l) surjects if/ 2. The study of the generators of the ring @lO H°(B-l) is closely related with the question of when B3 is normallv generated when B is mnple. ecall that a line bundle L is said to be normally generated, or to satisfy property ç. if it is very mnple and the image of the morphism induced by LI is a projectively normal variety. This is equivalent to the ring @0 H°(Le) being generated in degree 1. In the present context (X a Calabi-.u threefold and B an ample and base- point-free line bundle on X), the answer to the question of when B m is normallv generated is partially known. If the image of X by the morphism ç induced bv IBI is hot a variety of minimal degree, the authors gave a complete answer to this question for m 2 2: in this case, in [GP2], Corollm'v 1.1, Theorem 1.4 and Theorcln 2730 FtANCISCO JAVIEI GALLEGO AND BANGERE P PUtNAPtAJNA 1.7, they proved that B ®mis normally generated if rn _> 2. The way of proving those results serves to illustrate the relation between the study of the generators of =o H°(B®) and the nornlal generation of powers of B. For instance, in order to prove the llOrlnal generation of B 'v2, the first step is to show that the map H°(B e) ® H°(B :') H°(B 4) surjects. This was pr,,ve(l in [GP2] by showing that @ï--0 H°(B»t) is generated in degree less thml or equal o '2 and that the mai» H°(/3 ®3) ® H°(/3) -- H°(B 4) surjects. If (X) is a vm'ietv of mininlal degree, the situation is more conlplex. If m > 3. the authors also gave in [GP2], Corollary 1.1 and Theorem 1.4. a complete answer: B' is normally generated if and onlv if m > 4 or if m >_ 3 and ç does not map X 2 1 onto linear P:t. If m : 2, mlsv«ering the question will settle the following ' Conjecture 4.3 (cf. [(.P-I, Conjecture 1.9). Let X be a Calabi-lau threefold and let B bc an ample and base-poiat-free line bundle. Then B satisfles propert!/ No f and onl!/ if there is a smooth non-h!/perelliptic curve Cin lB ® Osl, for some s I1. This çonjecture would also give a characterization of when B ®2 is very ample. This question of v«hen B 2 is very ample is also open. One might ask what light the results proved in this section shed on the conjecture. Theorem 4.1 says that if (X) is Pa. then the ring @ï--0 H°(B®) is generated in degree less than or equal to 2 if and Olfly if the degree of is greater than 2. hl this case, however, it was seen in the proof of Theorenl 4.1 that ,a did hot surject. On the other hand. if ç(X) is a variety of minimal degree different from Pa, ,a does surject but t0 H°(B®t) is hot generated in degree less than or equal to 2. Therefore the strategy outlined belote to studv the normal generation of B ®2, which xvorked when (X) was hot of nlilfimal degree, does hot work if c2(X) is of minimal degree. We point out that the conjecture is nevertheless truc if (X) is linear Pa (sec [GP2], Corollary 1.8). This also follows from the methods of this article, by studying the map ,(2.2) in the proof of Theorem 4.1. Thus the onlv case left in order to settle the conjecture is when (X) is a variety of minimal degree different from Pa, which should he addressed using a subtler strategy. The results of Section 3 regarding the structure of the canonical morphisms of regular surfaces of general type onto surfaces of minimal degree has some interesting consequences for Clabi-Yml threefolds. As ve will see, Proposition 3.1. Crollary 3.2 and Proposition 3.3 prevent many a priori natural examples of Calabi-Yau threefolds from existing. This also shows that if there do exist examples of prime degree Clabi-Yau covers of threefolds of minimal degree induced by complete liuear series, then thev cannot corne from group actions. We smmnarize this in the next two corollm'ies: Corollary 4.4. Let X be a Calabi-Yau hreefold with ai wors canonical sia9ular- ities. Let B be a base-poin-free line bundle. Le X I" be the morphism induced bg the cornplete linear series lB[. Let n be he degree off. Let U be a smooh open se of I" whose complernea bas codimension 2. and let L be a liae bundle on U. Assue hat CANONICAL FIINGS OF COVEFIS OF SURFACES OF MINIMAL DEGREE 2731 If t" is a variet.q of minimal degree, then the degree 7 = 2 or n = 3. bt partcular, if is a c.,clic cover, the degree of ç is 2 or 3 Proof. Let t" be a suitable hyperplane section of I', and ,b' the lmllback of t" by ç. Then ç[s is the canonical nol-phisln of S and satisfies thc h.vpothcsis of Proposition 3.1, and the thesis is clear. [] Notation 4.5. We will call a lllOli)hislll sa.tisf.ving (*) iii the statcment of Corollary 4.4 a quasi-cyclic cover. Corollary 4.6. Let X be a Calabi-lau threefold with at worst caTonical sin9tdar- ities. If X --, l" is a 9enericallg finite mo7»hism oTto a smooth scroll l" C pN, then the degree of ç is even. In particular, there are no generically finite covers of odd degree of sznooth rational normal scrolls. Pro@ The proof is analogous to that of Corolla.ry 4.4. using now Proposition 3.3. hl [GP2] we described what finite lnorphisms frolll ri Calabi-Ym threefold onto a variety of minimal degree induced bv complete linear series were possible. After Corollary 4.4 and ('orollary 4.6 we can obtain the following shm'per version of the result in [GP2] (compare with [GP2], Proposition 1.6): Proposition 4.7. Let X be a smooth Calabi-lau threefold, let ç be the morphism induced bl the complete linear series of an ample and ba.se-point-free line bundle B on X. and let be the de9ree of ç. If ç(X) is a varict'g of miimal de9ree, then one of the .[ollowin 9 occurs: 1) t'=P3 andn<_24. 2) t is a smootb q.uadric hypersurface izt p4 and n = 2, 4.6.8, 10.12 or 14. 3) t" is a sznooth ratioTtal normal scroll of dimension 3 in P5 and = 2, 4.6, 8, 10 or 12. 4) Y is a sm.ootb ratioTtal normal scroll of dimension 3 i7 pN, - _> 6. n = 2. and X is flbered over P with a smooth K3 surface as a general fiber. The restriction of B to the general flber of X is hgperelliptic, witb sectional genus 2. and its complete linear series m.aps the fiber 2 1 onto a general fiber of the scroll. 5) Y is a smooth ratioTtal normal scroll of dimension 3 in PN N >_ 6. n = 6. and X is fibered over p1 with a smootb abelian surface as a general fiber. The restriction of B to tbe general fiber of X is a (1.3) polarizatiom and its complete linear series maps the fiber 6 " 1 onto a general fiber of the scroll. 6) Y is a cone over a cozic in pe. 7) Y is a coTe over a twisted cubic in p3. 8) Y is a cone over a Iéronese surface. In addition, if X bas at worst canonical singularities aTd ç is a quasi-c!Iclic cover, then n = 2 or 3. EFERENCES [Bo] E. Bombieri, Canonical models of surfaces of 9eneral type, Inst. Hautes Etudes Sci. Publ. Math. 42 (1973), 171-219. MR 47:6710 [Ca] F. Catanese, On the moduti spaces of surfaces of general type, J. Differential Geometry 19 (1984), 483-515. MR 86h:14031 [Ci] C. Ciliberto, Sui grado dei generatori detl'anello di una superficie di tipo generale, Flend. Sera. Mat. Univ. Politec. Torino 41 (1983), 83 111. M1R 86d:14036 2732 FRANCISCO JAVIER GALLEGO AND BANGERE P. PURNAPRAJNA [EH] [GPI] [a21 [GP3] [GP4] [GP5] [HI] [H2] [H4] l'<')al [Non] [MP] [M] [OP] [] D. Eisenbud and J. Harris, On varieties of minimal degree (a centennial accourir), Algebraic Geornetry, Bowdoin 1985, Amer. Math. Soc. Syrnpos. in Pure and Appl. Math. 46 (1987), 1-14. MR 89f:14042 F. J. Gallego and B. P. Purnaprajna, Projectwe normality and syzygies of algebraic sur- faces, J. Reine Angew. Math. 506 (1999), 14.5-180. MIR 2000a:13023; MR 2001b:13016 F. J. Gallego and B. P. Purnaprajna. Very ampleness and higher syzygies for Calabi-Yau threefolds, Math. Ann. 312 (1998), 133-149. MR 99g:14048 F. J. Gallego and B. P. Purnaprajna, Canonical covers of varieties of minimal degree, Preprint math.AG/0205010. To appear in "A tribute to Seshadri--a collection of papers on Geometry and Representation Theory", Hindustan Book Agency (India) Ltd. F..I. Gallego and B. P. Purnaprajna, Some homogeneous rmgs associated to finzte mor- phisms, Preprint. To appear in "Advances in Algebra and Geometry'" (Hyderabad Confer- cnce 2001). Hindustan Book Agency (India) Ltd. F..I. Gallego and B. P. Purnaprajna, On tbe rings of trigonal cu'ves, in preparation. M. L. Green. The canonical ring of a variety of general type, Duke Math..]. 49 (1982), 1087 1113. MR 84k:140(}6 D. Hahn and IR. Miranda, Quadruple covers of algebraic varieties, J. Algebraic Geom. 8 (19.q9), 1 30. MI/ 99k:14(}28 E. llori "kawa, Algebraic surfaces of general type with small cî. I. Ann. of Math. (2) 104 (1976), 357-387. MR 54:12789 E. tlorikawa, Algebratc surfaces of g«neral type with small cî, II. Invent. Math. 37 (1976), 121 155. IkIIR 57:334 E. llorikawa, Algcbraic sttrfaces of general type with small cî, III. Invcnt. Math. 47 (1978), 209-248. MR 80h:14012a E. Hori'kawa, Algebraic surfaces of general type wzth small cî. IV, Inveut. Math. 50 (1978/79), 103-128. MR 80h:14012b K. Kodaira, Plurzcanonical systems on algebraic surfaces of general type, J. Math. Soc. .]apan 20 (1968), 170-192. MR 37:212 K. Konno, Algebraic surfaces of general type with cî ---- 3pg - ô, Math. Ann. 290 (1991), 77 107. MR 92i:14039 NI. IVlendes Lopes and R. Pardini, Triple canonical surfaces of minimal degree, International J. Math. 11 (2000), 553-578. MR 2001h:14049 D. Murnford, Varzeties defined by quadratzc equations, Corso CIME in Questions on Alge- braic Varieties, Edizioni Cremonese, Rome (19ï0), 29-100. lkIl 44:209 K. Oguiso and T. Peternell, On polarized canonical Calabi-Yau threefolds. Math. Aire. 301 (1995), 237 248. MR 96b:14050 M. Reid. lnfinitesimal view of extending a hypeplane section--deformation theory and computer algebra, Algebraic geometry, Proceedings L'Aquila 1988. 214-286. Lecture Notes in Math. 1417. Springer-Verlag, Berlin. 1990. MIR 91h:14(}18 DEPARTAMENTO DE ALGEBRA, FACULTAD DE CIENOIAS ,.IATEM,TICAS. UNIVERSIDAD COM- PLt TENSE DE .MADnD. 28040 lklADnD. SPAN E-mail address : FJavier_Gallego@mat.ucm. es DEPARTMENT OF .IATHEMATICS. UNIVERSIT OF KANSAS. 405 SNOW H %LL. LAWRENCE. RANSAS 66045-2142 E-mail address: purnaath.ukans, edu TRANSACTIONS OF THE AMERICAN MATHEIvlATICAL SOCIETY Volume 355, Number 7, Pages 2733-2753 S 0002-9947(03)03272-0 Article eIectronicaIIy published on March 19, 2003 A CLASSIFICATION AND EXAMPLES OF RANK ONE CHAIN DOMAINS H. H. BRUNGS AND N. I. DUBROVIN ABSTRACT. A chain order ofa skew field D is a subring // of D so that d E D\R implies d -1 ( R. Such a ring R has tank one if J(R), the Jacobson radical of Ff, is its only nonzero completely prime ideal. We show that a railk one chain order of D is either invariant, in which case R corresponds to a real-valued valuation of D, or R is nearly simple, in which case R, J(R) and (0) are the only ideals of R, or R is exceptional in which case R coIltains a prime ideal Q that is not compleIely prime. \Ve use the group ,¥1(H) of divisorial R-i(leals of D with the subgroup 7-tIR) of principal R-ideals to charactcrize t hese cases. The exccptional case sul»(livides further inIo infinitely IIlany cases depending on the index k of 7-/(R) in J/(R). Using the covering group lE of SL(2, N) and Ihe result that the group ring TlE is einbeddahle into a skew tield for T a skew field, examples of tank one chain or(lers are const ructed for each possible exceptional case. INTRODUCTION A sul)ring R of a skew fit,ld D is called total if d in D and d llOt iii R ilnl)lies that the inverse d - is contained in R. It tbllows that for such rings 5' the lattice of right ideals as well as the latticc of left ideals is linearlv ordered bv inclusion: 5" is a chain dolnain. Conversely. ai13" chain donlaill is ()re and is a total subring of its skew field of quotients D. The total subrings of fields are exactlv valuation rings. corl'eSpollding to valuation fUllctions iiito lilmarly ordered groups. In particular, if we take nontrivial subgroups G of the additive group (IR, +, <_) of the reals as value groul)s , then we obtain the COlnnmtative vahmtion rings of ralik one. Such a ring Cml also be characterized as a niaxinlal subring of a field, or as a valuation ring with exactly one llOllZ(ro prilne ideal, hl the iion-commutative case we llillSt distinguish between prime ideals and coinpletely prime ideals: An ideal B - 5" of a ring 5" is prilne if Ii I.). C_ B ilnt)lies I C B or I _C B for ideals I and I of R. If ab B iniplies a / or b / for elements ri, b iii 5". then / is called completely prime. A total subring R of a skew field D will be called a chain dolnain of rank Olm if 5" has exactlv one llonzero COlnpletely prime ideal. This ideal will thell be J(,R), the Jacobson radical of R. Received by the editors April 10, 2002 and, in revised form. October 9, 2002. 2000 Mathematics Subject Classification. Primarv 16L30. 16K40. 16\V60: Secondary 20F29. 20F60. Key words and phrases. Exceptional chain domains, skew field, valuation, cone, covering group. The firsI author was supported by NSE1RC,. The second anthor was supported by RFBR and DFG (grant no. 98-01-04110). (2003 Anerican iMathematicaI Societ 3 ")73 2734 H. H BRUNGS AND N. I. DUBROVIN We prove iu Theorem 1.9 that a rank one chain domain /? is either invariant, i.e., all one-sided ideals are two-sided, or itis nearly silnple in which case H, .l(H), and (0) are its only ideals, or /ï' is exceptional in which case /? coutains a prime ideal that is not completely prilne. The exceptioual rank one chain domains are classified further with the help of the group Ad(H) of divisorial R-ideals and the subgroup 1-/(/) of Ad(/?) of principal/?-ideals. The lattice of two-sided/?-ideals is then determined by the index k of 1-/(/?) in Ad(/?), and we soEv that/? is exceptional of type These results are proved in the more general case of coims P in groups G where a cone P of G is a subsemigroup of G so that g E G\P iInplies 9 -1 E P. That rank one chain doInains are either invariant, nearly siinple or exceptional was proved in [4]. Invariant rank one chain orders of D correspond to valuation fimctions from D* into (IR, +, <_). Nearl,v simple chain domains were constructed in [8], [16], [5] and [3]. The construction of exceptional rank one chain domains, however, appeared to be elusive even though Posner in [19] hinted that such riugs might exist, and the classification of hypercyclic rings by Osofsky in [18] is complete only if su«h rings do not exist. I. N. Herstein had considered the problem and this existence problem was also encoulitered in [14]. Ve construct in this paper excep- tional rank one chain domains of anv type (Ce): TheoreIn 4.4 and Crollary 4.6. We do this by first constructing exceptional cones Pe of type (Ce) in subgroups H of the universal covering group G of SL(2,1R), Theorem 3.8, and then apply Dubrovin's result in [11], where he constructs ail exceptional rank oIm chain ring of type (C1) associated with a cone 1. CHAIN DOMAINS AND CONES 1.1. Basic properties. A ring R is a right chain ring, if the set of all right ideals of R is linearly ordcred with respect to inclusion. Left chain riugs and chain rings are defined silnilarly. A chaiu dolnain R has a classical skew field of quotieuts D aud can therefore be considered as a total subring of D ([7]). A subsemigroup P of a group G is called a cone of G if G = P U p-1 and P is a pure cone if in addition P f3 p-1 = {e}. There is a close commction between cones Pinagroup G and right orleft orders: if Pis acone of G and a,b then _ < defined by a _ < b if and only if a-lb P defines a left preorder, and a<_b if and only if ba - P defines a right preorder on G. The relations "<_" and ê are right orders and left orders on C, respectively if and only if the cone P is pure. Fiually, if P is pure, then the right order defined by P agrees with the left order defined by P if and ouly if aP = Pa for all a in C,, i.e., P is invariant under all imper automorphislns of C,. The group C, is then linearly ordered. Let P be a coue of a group G. A nolmmpty subset I of G is called a left P-ideM if PI c_ I and I c_ Pe for a suitable elemeut a in G. The second condition is satisfied for anv I ¢- C, provided I satisfies the first condition. If in addition I C P, we say I is a left ideal. Flight P-ideals, P-ideals and right ideals and ideals are defined similarly. An ideal B of P is called a prime ideal if B -¢ P and aPb C B inplies a B or b E B for a, b P. If ab B implies a B or b B for the ideal B -J= P of P, then B is called completely prime. We collect elementary properties of a cone P in G. We tan assume that P -J= G. Let U(P) = P fil p-l, the subgroup of units of P. CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2735 a): J(P) = P\U(P) is the nlaxilnal right and the nlaxilnal left ideal of P: it is the Jacobson radical of P and it is a colnpletely prime ideal of P. b): The set of right (left) P-ideals in G is linearly ordered with respect to inclusion. \Ve define I _< 12 if and ouly if I _D I2 for right P-ideals I and I» To sec this, one considers fil'St principal right P-ideals oP and bP in G. Then either a-b P and bP C aP or b-la P and oP C bP. If 1.2 Il, then there exists a in I\I and Il C oP C I., follows. c): There is a one-to-one corresi)ondence between the set of COlleS P' ¢ G in (7, that contaill a COllP P and the set of completely pl'illlP ideals B of P. Pro@ Let P ç P' C (7 be concs in G. Then j' ¢ J(P') and j' P implies j,-1 ¢ p, a contradiction. Hence, J(P') Ç J(P) and P' = P U (P\J(P'))-. Converselv. if B C J(P) is a completely prime ideal in P, then P' -= P U (P\B) - is a cone 7 (7, in G. [] d): Let I be an ideal in P with I 7 P and Q = Ç]I '' 7 0. Then Q is a «olnpletely prime ideal. Pro@ If c G P\Q and ca G Q for somc o in P, then there exists n0 with c I'°. However, for any n there exist ai, bi G I with ca = a...a,ob.., b,,. Then a...a., o =cdforsonledin Pand a=db...b, G I' follows. Hcnce, aG Qand Q is a completely prime ideal. [] e): A P-ideal I will be right principal and lefl principal if and only if I = zP = Pz for some z ¢ G. Pro@ Let I = zP = Pz with zl,z G. Then z = zla, z = bz fol some a, b P. Hen«e, bza = z. Since I is an ideal, there exists b' in P with bz = zb' and z = zb'a follows. Therefore, b'a = 1 and a U(P), and Pz = zP = ZloP = z P. [] f): Let P be a cone in G. The set "H(P) of all principal P-ideals of G forms a group with ideal multiplication as the operation. "H(P) is isomorphic to a subgroup of (IR, +, _<) if J(P) is the Olfl.v completely prilne ideal of P. Pro@ If I = zP = Pz and I = zP = Pz, then ltI2 = zPz,P = zzP and (zp)- - - -- ,, P. It follows that 7-/(P) is a group with P as identit.v. To prove the second statelnent let P D zP = Pz. Then Ç](zP) = since otherwise Ç](zP) is a completely prime ideal - J(P) by d). "H(P) is therefore an ordered Archimedean group and the statement follows from H61der's Theorem (see [13]). [] g): A right P-ideal I is a principal right P-ideal if and OlflV if IJ(P) ¢- I. Pro@ If I = zP, then zP D IJ(P) = zJ(P). Conversely, if I is hot principal as a right P-ideal, then for a ¢ I there exists b G I with «P C bP, = bj IJ(P), j ¢ J(P), and IJ(P) = I. [] We single out cones with the property in f)" Definition 1.1. A cone P of a group G ha.s tank one if J(P) is the onlv completely prime ideal of P. 2736 H.H. BRUNGS AND N. I. DUBROVIN It follows flore the definitions that a subring R of a skew field Dis total if and only if the semigroup R* = (R\{0},-) is a cone in the group D*. This relationship between a cone in a group and a chain domain is generalized in the next definition. Definition 1.2. A total subring R in a skew field D is said to be associated with a cone P in a group Gif the following conditions hold: i): G is a subgroup of D*, the multiplicative group of D. ii): Ever.v element d in D* tan be written as d = gal = 292 with g.g2 in G and Ul.«2 in U(R) so that PglP = PgaP. iii): RC G = P. We also say in this case that the cone P is associated with the chain domain R. Proposition 1.3. Let the total subrbtg R of the skea, field D be associated with the coe P of the groap G. Then: i): Io --> loR defitcs an isomorphimn front the lattice of right P-ideals fo the lattiçe of nonzero right R-ideals. The im,erse of this rnappmg assigns I C G lo the nonzero right R-idcal I. ii): The correspmMeucc dcfincd in i) preservcs the properties of being an ideal. a completel9 prive ideal, a prime idcal, and a principal right ideal. Pro@ i) If I0 is a right P-ideal. then tvo form a = 9u, b = g2tt2 for gi I0 and ui and gl = gP. P G P follows. Therefore, a that loR is a right R-ideM. since gI C_ P nonzero elements a. b in loR have the U(/). We can assume that g P C_ g2P, -t--b = g2(ptq -t--"a.2) /oR: this shows C_ R for some g G C D. Further, if g IoRçG for a It follows that hg Similarly, one can (I n G)R = I. right P-ideal Io, then 9 = ¢I0anduCU(R) GG= show that I VI G is a right hg'u for h L» g' Ç P and n U(R). U(P); hence, 9 Io and loftG = Io. P-ideal if I is a right R-ideal and that For ii) we only show that the right P-ideal I0 is a P-ideal if and only if I0R is an B-ideal. Let r R and h Io, a P-ideal. Then r = plu for p P, U() and rh : plulh = pL'u2 tbr ulh = L'u2 with u2 LT(B) and h.k G. Bv ii) of Definition 1.2 we have PhP = Pi'P; k Io %llows and rh Io, which shows that I0B is also a lefl B-module and then an B-ideM. Comrsely. if IoB is an B-ideal %r a right P-ideal 10, then I o = IoB G is a P-ideal. Some variations of the results in this section tan be round in [12] and [6]. 1.2. Divisorial ideals. consider certain P-ideals for a cone P which will fornl a group in case P has tank one. Definition 1.4. Let P be a cone in a group G. The divisorial closare Ï of a right P-ideal I is the intersection of all principal right P-ideals containing I : ï: hPDI A right P-ideM I is called divisorial if Ï = I. If we replace the cone P by a total subring R. we obtain the definition of the divisorial closure of a right R-ideal and of a divisorial right R-ideM. In addition. we assume that a divisorial right H-ideal is nonzero. b collect a list of properties: CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2737 Let P be a COlle in a group G, I a P-right ideal. Then: a): Ï_D I; b): I = I: c): gI = gÎfor any g in G: d): I is non-divisorial if and only if J(P) is hot a principal right ideal and there exists ail elelnent z in G with Î= zP and I = z,l(P). If, in addition. I is a P-ideal and rank P = 1, then Ï= zP = Pz and I = zJ(P) = J(P)z. The properties a, b, and c follow directly froln the definition. To prove d) we will write J instead of .l(P) and assume that ï D I aud that z C Ï\I. Then î zP D I and Ï = zP follows; then I = z J, since zjP I for some j .I(P) leads to a contradiction. This also shows that .1 is hot a principal right ideM. If J is not a principal right ideM, then cP zd implies z-«P d and z-«P P, cP zP for c,z G. This means that Ï = zP for I = zd and hence Ï D I. If zP is a P-ideal, then certainly zJ is a P-ideal. Conversely, if zJ is ail ideal, tllen zP is an ideal, since otherwise there is an a P and a j .l with azj = z, a contradiction. Finally, we asslillle that Ï # I and I is a P-ideal alld that P has rank ont'. Tllen ï = zP and the left order O((I) = {.q e G J gï ç ï} # G «ontains the cone P as well as the cone zPz - both of which are maxinlal. It fifllows that P = zPz -, Pz = zP = Ï and .lz = z,l = I. b list a property that was proved in the proof of d): e): I is a P-ideal if and oldy if I is a P-ideal. The next result shows that in the correspondence between right R-ideals and right P-ideals, divisorial right ideals correspond to each other if the chain domain R is associated with the COlle P. Proposition 1.5. Let R be a total subrin9 of the skew field D associated with the cone P irt a group G. Then the right P-ideal I is divisorial if and only if the right /'-ideal I B is divisorial. Pro@ Assume I is divisorial, with hP D_ I, and i.e., I= Ï= Ç hP. Then I/i'ç h/i'for all h cG hPDI hR. To show the reverse inclusion, let d ÇI hB hPDI for hP D_ I and d = hr h = gin fol g lâ. m U(/). HellCe, 9 = hrh m-1 ¢ hHG = hP and 9 hP = I. d IB follows. Now assulne that .4 is a divisorial right R-ideM. A = dR. Any Stleh d = gin fbl'g G G, II G U(). Hen«e, dRA AaG = (dR)G = (gRG) = gP. which shows that 4aG is divisorial and A G is nonempty, since .4 is nollzero. For any subset I of a group (7, we define the following three subsets of G : the right order O(I) = {9 e G [ 19 C I}, the left order O«(I) = {9 e G [9I ç I}, and the inverse I - = {9 G I IgI ç I}. It follows that I - = {9 e G 191 ç O,.(I)} = {9 e G I I9 ç O«(I)}. We have the following two properties where P is a cone in the group G : f): If I is a right P-ideM. then Or(I) is a COlle of G and O,.(I) is an over cone of P. Fnrther, I is a right O(I)-ideal and a left Or(I)-ideal. and i-1 is a right Or(I)-ideal and a lefl. O(I)-ideal. 2738 H. H. BRUNGS AND N. I. DUBROVIN For a proof we observe that for any g in G either gI C I and g 0(I) or I C gI and g-1 Oe(I). The test of the statements follow inmmdiately. g): O,.(J(P)) = O«(.I(P)) = P, and j(p)2 ¢ j(p) implies that J(P) = zP = Pz for solne z P. The first statcment follows froln Property c) iii Section 1.1 since Or(.](P)) D P implies that j-lj(p) Ç .I(P) for some j .I(P). Hence, .I(P) Ç jJ(P), a contradiction that shows Or (J(P)) = P and similarly O/(.I(P)) = P. The second statenmnt follows from Propcrty g) in Section 1.1, its left S,Vlnnmtric version, and Property e) iii Section 1.1. [] Even though one can consider the groupoid of all divisorial P-ideals for a cone P of arbitrary rank (see also [2]), we restrict ourselves to the rank one case: Definition 1.6. Let P be a cone of rank one. Then .Ad(P) is the set of all divisorial P-ideals togetller with the operation " Il * I2 = Il I2 \Ve have the following rcsult: * " defined by: foL P-ideals 11, I2. Theoreln 1.7. Let P be a cone of rak oe in a group G. Tben: ) ./( P) is a linearl!! odered gvup: d) The iv«rse of an elemett I in .Ad(P) is ) (P) is a subgro«p of.Ad(P). Pro@ We show first that the operation defined in Definition 1.6 is associative. On the set of ail P-ideals we define a relation IL I2 if and only if Ï = Ï2; this is an equivalence relation. We are going to show next that for P-ideals Il. 1"2 the following equivalence holds: (+) II2 ,-, 1112. If I1 = Iî and I2 = Ï, thell (q-) iS trivially true. If 11 Iî, then Ï = zP D zJ(P) = Il and J = ,J(P) is hot right principal. Also Ïl = zP = Pz is a P-ideal by Property d). The equivalence (+) holds therefore if and only if the following equivalence holds: (++) JI2 ' PÏ2 = Hence, if JI2 = I2, we are done. Otherwise, JI2 C 12 and 12 = Pd follows for some d in C, by the left synlmetric version of Property g) in Section 1.1. Since I2 is an ideal, we have dP ç Pd, P ç d-lpd and the equalit.v d-lpd = 19 since P has rank one. Then dP = Pd = 12, dJ = Jd and JI 2 = Jd = dJ , dP = I2 which proves the equivalence (++) and hence also (+) in this case. Finally, we must prove (+) if I1 = Î1 and Ï D 12. Then, as above, ï2 = aP = Pa D aJ = Ja = I2 for sortie a in G. The equivalence (+) then holds if and only if the equivalence I1J -- Ï1P = 11 holds. Using the right symmetric version of arguments used in the proof of (++), one shows that I1J ' I. This proves (+). If 11 -" I and 12 I for P-ideals I, I, 12, I, then 1112 Ïl Ï2 = 1112' g' ILI .' ' Hence ./bi(P) is a factor lnonoid of the nlonoid of ail P-ideals, and the operation given in the definition for .Ad(P) is associative. Next we show that II - = P for I a P-ideM, and I-I = P follows from similar arguments. Since I is a P-ideM, 1-1 is a P-ideal. CLASSIFICATION OF RANIx ONE CHAIN DOMAINS 2739 If I1-1 = P, we are done; otherwise I1-1 C_ J(P) = J. If I1-1 C_ Pz C J for some z E J, then II-lz -1 C P and I-lz -1 1-1 which ilnplies z P, since P has rank one. This is a contradiction since z E ,1, and 11-1 = ,1, ,1 7 Pz for all z E P remains as the only l»ossibility to be considered. It then t'ollows ff'oto Property g) that ,l 2 = J, J is hot a principal right ideal, and hence I1-1 = ,Ï= P. In order to complete the proof of o) and /) we show that I - is a divisorial P-ideal for I a P-ideal. If on the contrary, I - = zJ C zP =- 1-1 and J is hot a principal right ideal, then zJI C_ P b.v the definition of i-1, and l»y (+) it. follows that zJI C_ P = P. Since zJ = zP, we obtain z.l C_ zÎc P, and hence z E I 1 = zJ, a contradiction. This shows that .Ad(P) is a group and thal /3) holds. For 11 _D I2 in .Ad(P) we define I < I. and .Ad(P) then is a linearly ordered group with P as identity. Elelnents in ?-t(P) have the form I = zP = Pz for some z in G with zP = and (zP) -1 = z-lP = Pz-1; see f) in Section 1.1 and ï/) follows. This proves the theorem. [] Corollary 1.8. Let P be a cone of ratk one in a group G. Thet ,(P) and (P) are A rch ira edean gro ups. Pro@ Let B C P be a divisorial idcal. IfB C J(P) = Jor B = J ¢ J2, then ÇB n = 0 by Pl'operty d) in Section 1.1. If J = ,I ', we have ,I = P and henee Ç] B' = 0 in all cases, and B'+ C B'. Then ,+1 C_ B', since there are no further right ideals between B'+1 and /,+1. This illlplies Ç/' = O, and it follov, s that .Ad(P) and 7-{(P) are Archimedean: see also Property f) in Section 1.1. [] Related results can be round in [12] and [2]. 1.3. The classification of tank one cones. The groups fld(P) and ?-/(P) will be used to classify tank one cones P in groups G based on the lattice of their ideals. In the following theorem and proof we will write J instead of J(P). Theorem 1.9. Let P be a cone of rank one in a group G. Then exactly one of the following possibilities occurs: A) - The cone P is Archimedean. i.e., aP = Pa for ail a in P. IVe distinguish two possibilities in this case: Al): fl//(P) = -(P) (Z, +, _<). tt,hi«h is exactlg the case when J ¢ J Then every P-ideal is a power of J and the cone is called discrete. A2): .Ad(P) (IR, +, _<) and ?-t(P) is a dense subgroup of ,M(P). B): The cone P is nearly simple; i.e., J is the only proper ideal in P. In this case 34(P) = (P) = {P}. C): The cone P is exceptional; i.e., there exists a prime ideal Q in P that is not completely prime. Then: i): There are no further ideals between J and (2. ii): The ideal Q is divisorial and .Ad(P) = gr {Q} is an infinite cyclic group. iii): Q" = O. iv): There eists an integer k >_ 0 such that "H(P) = gr{Qk}. The cone P is said fo be of type (C) in this case If P is of type (C0), then " D (Q)-I D...DQ - DPDJ DQDQ DQ 3 D-.- 2740 H. H. BRUNGS AND N. I. DUBROVIN is the chain of P-ideals. If P is of type (CI), then -" D Q-2 = z-2P D z-2J D (2 -1 = z-IP D z-l.] D P D J D zP = Q D zJ » z2P = 0 2 D z2J D ... is the chain of P-ideals. If P is of type (Ck). k > 2. then "'" D (Qk+I)-I D z-IPD z-lJ D (Qk-1)-I D"" D Q-1 D PD J D Q D Q" D ... D D zP D zJ = 0 h" D Qk+l D "'" D Q2k-1 D z2P D z'J = (-2 2t D Q.t+l D "'" is the «hain of all P-ideals. Pro@ If J is the olllv proper ideal of P, then P is of type/3. Otherwise, let Q = U I be the lllliOll of ideals of P properly contained in J. If .1 : = J and J D Q, then P is exceptional: for ideals I D Q and I2 D Q in P we ha'ce If . I2 D_ j2 = ,1 D Q and Q is a prime ideal of P. hot completely prime and no fmther idem exists between ,I and Q. The divisorial closure Q of Q is ail ideal that CallllOt be eqllal to ,1, since J would then be right i)rillcipal. Hen«e, Q = Q is the sma.l]est positive element iii the linearly ordered Archimedean group ,M(P), and .Ad(P) = gr{Q} is ail infinite cy«li« group. The subgroup (P) has therefore the form (P) = gr{Q t} tbr sonle k _> 0: we sav that P is of type (Ck). We can now des«ribe the P-ideals in ea«h case (C) if we recall (sec Property d) in Section 1.2) that ail ideal I is either divisorial or of the form c,l = Je with Ï = oP = Pe. some c ¢ G and J = J-. It will also follow fl-Olll the rest of the proof that if P is exceptional, then J = j2 and J D ( = U I. where the ideals I are properly COlltailled iii J, the prime ideal that is llOt completely prime. In the case (C0) there are no principal ideals ¢ P and the group ,l(P) = gr{Q} «ontains all P-ideals ¢ J. In the case ((71) the ideal Q = zP = Pz is principal and .Ad(P) = (P). In the case (C), k _> 2. the ideal Q is principal. However. Qh itself cmmot be principal, sin«e otherwise Q = zP implies QhJ ¢,Qh; hen«e, Q,I ¢ Q and Q is principal (sec Property g) in Section 1.1). Hence Qh = zP = Pz D Qk = zJ = Jz for an element z in P. It remains to consider the case where either J ¢ J, j2 or J = and J = Q = UI for ideals I properly contained iii J. hl this case we will prove that aP = Pa for all a in P. If for some a in P the right ideal aP is hot a left ideal, then an element c exists iii P with cap D aP and caj = a follows for ail element j in J. Bv assmnption there exists an ideal I C_ J with j ¢ I and ["] I n = : we obtain the contradiction a = caj = cnaj Ç]I n . We have Pa ç aP, P c_ aPa -1 and P -= aPa -1 since P is of rank one. Therefore, Pa = aP for all a in P and P is invariant. If J =/: J', then J = aP = Pa, fol" some a in P, is the smallest positive element in the Ar«himedean group .Ad(P). Hence, ,4(P) = (P) = gr{J} is the group of all P-ideals. If J = j2 and J = Q. then (P) is isomorphic to a dense subgroup of (IR, +, _<) and .Ad(P) is isomorphic to (IR, + _<). [] If R is a «hain order of rank one in a skew field D. then R* = R\{0} is a cone in the group D*. We say that R has type (A). (Al), (A). (B), (C), or (C) if and onlv if the cone R* is of the saine type. CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2741 The next result follows fl'oln Prol)ositions 1.3 a.nd 1.5 and Theoreln 1.9. Corollary 1.10. Let P be a coe asso«iated witb the rank oe cbai domaiu R. The'n P and R bave the saine type. 2. THE UNIVESAL COVEmNG GROUP OF SL(2,) 2.1. The group SL(2,). By SL(2,) we denote, as usual, the group of 2 x 2 lnatrices with l'cal entries and deterlnilmnt equal to 1. Then U= = 0 a - and 3 = r(t) = Sill t COS t t ] are two particular subgroups of G. Every C|elm,nt s SL(2, ]) ca.n be written in a Ulfiquc wa" as s=r(l)l, for r(t) a with 0t <2 and u . To prove this clailn, let {e, e2} be the standard basis of 2, the Euclidean plane, and let the elelnents of SL(2, ) be the lepl'eSeltatiols of linear trans%l'lnations of 2 with respect to the basis {ci, e}. For every nonzero vector a there exists a unique element t e [0, 2) with a/llall = el cos t + e2 siu t: we write arg a = t iii this case. Let t = arg S(el) for the givell elelnellt 8 SL(2, SOlne elelnent v. since r(-t)s(cl) = ael for o > 0. representation is tllliqlle, Sillce lJ Ç/ ---- { [}. I -- ( ï ) 2.2. The group G. We are going to constrllct the ll) and r(-t)s = u IL ! fol" Hence, s = r(t)u and this the identity of SL(2,1R). Ulfiversal coveling group G of the group SL(2,1R) in this section. \Ve do this first fOl" the subgroup bv fixing a sylnbol, say x, and bv rewriting the additive group of the rem munbers ill lnultiplicative fOl'lll: R = {x t I t ll}; X t' a "t= = Xt'+t2: x t' Then R is a linearlv ordered group isolnorphic to R to g with r(x t) = r(t) is a group epilliorphisni with <_ x t2 z;îe t l _ t2. +, <_). The mapping r frolll the cyclic subgroup gr{x 2r } as its kernel: r is a cover of the Lie group g. Next we define the covering group G of SL(2, IR) as the set G = {xtu I xt R, U}, the Cartesian product R × U, together with the following operation: If x t Ul, xt2zt2 are two elelnents in G and tu = 2rrk + ç for k Z and ç [0,2rv), then ulr(ç)u = r(b), in SL(2.1R) for u U, b [0, 2rr), and the product in G is defilmd as xtlUl -xt2u = The mapping r from above can be extended to a lnapping froln G to SL(2, by defilfing (") = ,-(t). We want to prove that G is a group and that r is an epilnorphism from G onto SL(2,IR). Lemma 2.1. The mapping r is onto SL(2, IR). and if a b = c for elemerts o, b. c in G, then r(c) = r(a)r(b). 2742 H.H. BFIUNGS AND N. I. DUB1ROVIN Pro@ The element a.t u in G satisfies v-(.rtu) = r(t)u for the arbitrarv element r(t)u in SL(2, OE); ç is onto. Ira = zUl, b = z»u in G, t = 2k+ç, k G Z, ç G [0.2) and if ur(ç)u = r()u. [0, 2). ui, u U, then c = .rt++'u and = (0)(2 + ç) = ()(), which proves the lclnlna.. Several special cases of the associative law fol" the operation defined for G are proved in the next few steps. We can consider R as well as U as subgroups of G ad the equations (+) .r t . u = ad'u, .r t-'tu=.r+tu, and .rtu-u'=xtzu ' fifl|ow. We conç|ude also that .r t a = .r t b imp|ies a = b for elements a. b G G. Lemma 2.2. For any element g = xtu G atd any m Z the product g..'' is cqtml fo Pro@ We have 7rm = 2rck + 99 with/," G Z, and = 0 if m is even, and = 7r if m is odd. In both cases ur(ç) = r(ç)u follows, which proves the statelnent of the lemma. [] Lemma 2.3. For an.q a, b G and ang integer m Z the following equalities hold: .m. («. b) = (.<"- ). b = . («m. b). Pro@ Because of (+) the first equation follows, and we can assulne that a = tt U and b = ..t ¢ R in the second equation. It remains to prove the following equality: («m. ,,). = . (m. ) where t = 2k + ç, k Z, ç [0, 2) and ur(ç) = r()u' for [0.2), t U. Then (.rm.u).t = zm++g'U '. distinguish three cases in order to compute the right-hmd side of the above equation. In the first case, m = 2k' is even and the equality follows ilnlnediately. In the sec(md case, wm = 2wk' + w for some k' G Z and < . Then tf .T 2(k+k )++ .2(k+k )tf - 3 "+ .2(k+k')++g.gt since ur(g + ) = lll'(g)T() = T(g)7"()lt t : T(g + )U t in SL(2, N); the equation is proved in this case. hl the final case, m = 2k' + for U G Z and ç . The right-hand side of the above equation is then equal to U " "2(k+kt+l)+- W(k+kt+l)--+llt which proves the lelnlna. Lemn,a 2.4. Letu G U andt (Trm, Tr(m+l)) for sornem Z. Then u.x t = x t u o.' ç t' (,.(,+ )). Pro@ Let t = 2rrk+ç for k G Z, q G (0,2rr). If rn = 2k is even, then G (0, rr); hence sinç > 0. It follows that for any u = (0b_l) U; the argulnent a cos /, of //r(ç)(el) = r())ttt(el) is also il, (0, rr) since b = a.rg [( 0 o, b-l ) ( SiYl(t'9 )] and a - sin ç > 0. Hence t' = 2rrk + ç G (Trm, 7r(m + 1)) as stated in the lelnma. CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2743 If rn = 1 + 2k is odd, then t = 2rk + with ur(ç) = r()'u; is also negative with the above argmnent; hence, (, 2) and t' = 2uk + Theorem 2.5. a): is a group; b): The mapping r is a homomorphism frvm onto SL(2, ); c): The center of G is the i¢nile cyclic group gcnerated by x . Pmof. To show that the operation defined for G is associative we consider three elements xt'ui G, i = 1, 2, 3 with ri and ui and thc equation By Lemmas 2.2 and 2.3 this equation holds if and onlv if the following equation is true: (x"+ " *+)" x+" =/'+%" (*+%" *+a) for integers k, m and n. It follows that it is suffiçient to prove (.) only in the case where t, t2, t3 [0, ). For g = x«u ' and g2 = x 'u with t, t' N, ', " we apply Lenmm 2.1 and obtain ,-(t)' = (,'(ri).1-T(t2),,2)- T(t3),3 and in SL(2,1R) where the operation is associat.ive, and therefore r(t)u' = r(t')u" follows. This implies u' = u" and t - t' = 2rrk for sonle " Z. It remains to show that W apply Lemma 2.4 ad obtai z «z for z U, [0, ); z «ta = «t3 for U, t3 e [0, n); u2x ta = xt3 for u 2 U, t 3 [0.); and l-t2+t : .çt2'3ï for uï Z ç and t2,3 [0, N). Therefore: gl = (XttUl " Xt2t2) " Xt3U3 = '{xtt+t'U'l U2] " Z3U3 and g2 XttUl (3:t2t2 :Et3t3) xtltl 3;t2+t'3 t : . : 23 t+t2 3- Il I = " ' "1 23" Hence, t = t + t + 3 and t' = ri + t2,3 and therefore t-t'= ' t2 + 3 - t2,3 = 2k. However, t + 3 and t2,3 both belong to [0, 2) and k = 0 and the associative law follows for the operation defined for . Since bas e = x°E, for E = ( ï), as the identity and xtu bas u-x -t as its inverse, is indeed a group; this proves a). The statenlent b) was proved in Lemma 2.1. It follows from Lenmla 2.3 that gr {x } is contained in the center Z() of . Conversely, if xtu Z() for t and u U, then an application of Lemma 2.1 shows that r(t)u is in Z(SL(2,)). 2744 H. H. BRUNGS AND N. I. DUBROVIN Ht'nce r(t)u = +r(0), u = ( ï), and t = 7rk tbr some k e Z follows. Therefore a- gr{x}, which proves c) and the theorem. See also [1] for the fact that G is right orderable, but hot locally indicable. 2.3. The representation of the group G. To each element 9 = xta G we can assign the projection v(9) = v(x t u) = . The nmpping I G Aut (, 5) is defined as 1(« ) = v(9 xt) for9 G, x t R. That 1 is indeed an automorphism of (R, ) folh)ws from the next result. Lemna 2.6. For g G let l be dçfined as above. Then: b): 1 is th« id«ntity mapping if and only if g is the identity element in . ): T,« .bim« t(') = {g e (-') = .,-'} i « o *çx - V, wbich is an Ore group. d): 1 is an automorphism of (R, ) for «v«ry g . Proof. To prove a) we comlmte v(ggx t) and ,,(ge,(g2x*)). Let g = x t' ai, g2 = a:t2u2 for ai U. Then gig2x xtux2a2xt xtuixt2x t u for some U, ' with u2,r t = s.t' u'. Furthcr, a.t ua.t2+t' u' = xt'+ï u' for ux t2+t' = xFù fl,r U, ï . It follows that v(gg.2x ) = a: '+ï and that v(gv(g2xt)) = v(xt, u.r2+t') = t,+ï; this proves a). To prove b), assmnc g = .t u and l(x t) = x t for all t . For t = 0 it follows and assume that u ). Then (x) = x that t = O. X consider t = = (ô implies that r[( l)(ï 1)()] :r(l): . , ). Finallv. for t = we must have Hence. b=0and u=(ô . arg = - and a a-a 1 follows: hence, 9 e, the identity in , and b) follows. To prove c) we observe that st(x°) = {x tu, G ] ç(z0) = wt = } equals U. Hence, I(x t) = xtU -* U. These stabilizers are Ore groups in the sense that the group ring TU over a skew field T is an Ore domain. This is true since U is the semidirect product of the following two torsion free abelian groups: This proves c). Finally, we want to prove d). Since l is an atttomorphism of (R, ), it follows from a) that it is enough to show that ç is an automorphism of (R, N) for any u V. Ve show first that z t > z t implies I(z t) > t,Ç( ) which then implies that l is one-to-one and order-preserving. Bv Lemlna 2.4 and Theorem 2.5(c) we can assume ri, f2 [0. g). It then follows that - f (-, ), and in addition f - > 0 if and only if ( cos t cos t2 sin(t-¢)= Det,sint sint) >0" hv (t) = (d) d [0.) ,d d = g(,(')) [0.). Th, sin t, Det (u(C°s t, cos t )) > 0, since Det (u) > 0, and, as in the previous argument. sin t sin t2 CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2745 de > dl follows. This shows lhat x d2 = l;(a "t2) > ..d = lç(x h) for t2 > ri and that IÇ is order-preserving and one-to-one. It remains to show that 1 is onto, and by Lemma 2.4 and Theorem 2.5(c) it is enough to show that 1" maps the interval [x °,.r '] onto the interval [x °. x']. This, however, follows from the fact that l(a "°) = x °, l(z ') = x' and that 1" is continuous. [] We will prove next a technical result, which will be used several times. Lemma 2.7. Let g = ztu G with t = 7ch + fo and x q R with t = 7cm +/10 for k, m Z and fo, tlo [0, 7c)..4ssume that ( ) ]I 2 ?.t, ith arg ( ) = tlo. Then l(,r h) = x rr(k+m)+t' for t' = arg (r(to)u( ; )). Pro@ Bv definition we have that l(a "t' ) = u(9a 31). Further, «{1 = OE.'{L-'I = *(k+m)zt°'tza'h° since z is in the tenter of G by Theorcm 2.5(c). By Lemma 2.4 we have ,«xt° = x i wit h G U alld ï= arg (ur(tl0)( )) e [0. g). Hence, z'u.r t'° = x'°+tù. ()n the other hand, t' = arg(r(t0)u(;)) = t0 + since both t0, ï [0,). It fillows that 9.r t = a-(k+)+t°+ia and l(3 "ri) = .r(k +m) +t' . 3. EXCEPTIONAL CONES IN THE UNIVERSAL COVERING GROUP In this section we construct exceptional cones of type (C) fol" everv k in the universal covering group We define first two particular elements Wl, tt' in G which will play m important role in this construction. The element Wl = ( î) e U C and r(wt) = a'l tbllows. Next we consider the elemcnt ( (0, w) and define we where u = r(-a)( ï) e U; hence, r(we) = ( ï). Lemma 3.1. Let b be an elemet in [0, u). The lira ç;»ï(z b) = x °. Pro@ We consider the real mmfl»er b.,, with x b = V (xb). Since u,ï = ( ) and r(wî) = wî, we can apply Lemma 2.7 and obtain bn = arg [( n ) (cosî)] = arg(cosb+2nsinb) sin sin b " If b = 0, then b = 0 for all n 0 and the result follows. If b (0. ). then sin b > 0 and linl (cosb are now read3 to define one of the main objects of this paper: e = { e ç [ (.0) 0}. The next result shows that this is an exceptional cone of type (C 1) in Theorem 3.2. a): ThesetF={ge[ I(x °) Rx°}isaconein u,ithU(F)= U. b): Any right F-ideal is either a principal right ideal xtF or of the form xJ() for some t d): The cone P is exceptional of tank one with Q = «P the prie ideal that is hot completely prime; is exceptional qf type (C). 2746 H. H. BRUNGS AND N. I. DUBROVIN Pro@ a) If g and h are eleinents in IP, then th(:r °) = V9(Vh(:r°)) _> ç(x °) OE 0 bv Lemina 2.6 a) and d), and gk follows. If g is not in P, then lÇ(x °) < x°; hence, x ° < I'- (x °) again by Lemma 2.6. and g- follows and is a cone of G. It also follows from the above arguments that g, g- implies I(x °) = x ° and g U. Conversely, U C and U() = U follows. Hen«e, J() = {g e G I;(x °) > .r°/. b) Let I be any right -ideal in G. Then it follows that x = iiff{ IÇ (x °) g e I} exists since I Cc for soine c G. çk will show that Ï = x for the divisorial closure Ï of I, see Definition 1.4. By definition we have x g = xç for ail g I since a N ff; hence x I. Conversely, if h G with h = xT I. then 7 ç(x°) for ail g in I and 7 a follows; hence Ï = x. It follows that either I = z = ïor that Ï= x and I = x%/(P); see Property d) in Section 1.2. c) Assulne that zt is a -ideal. For t = m + t0, m Z and t0 [0,) it follows that x° is also a -ideal since x is central in G. If t0 > 0, it follows froin Leinina 3.I that there exists a power wï of w in U with wïxt° D xt°, a coixti'adiction that shows that xt = xm. If I = xtJ(P) is a -ideal, then Ï = xt is a -ideal by Property d) in Section 1.2, and t = mn by the above arguinent. d) We have D J() D x = Q and Q is hot a coinpletely priine ideal of , since z /- x / Q, but z / Q. However. Q is a prime ideM, since anv ideals A and B of P that contain Q properly, also contain J(); hence. AB J()J() = J() D Q, and it follows that Q is a prime ideal that is hot completely priine. There are no firther ideals between J() and Q. and Q = . It follows that is an exceptional cone of type (C1): see Theoreii 1.9. We denote by F the subgroup gr{wl,w2} of G generated by u,1 and w2. This subgroup is mapped by - onto the subgroup gr { ( î), ( z ï) } of SL(2, N) generated by ( î) asd (z ï). Since this subgroup of SL(2, N) is free (see [I5], 14.2.I), the group F is free of tank 2. It follows [roln Lemma 2.7 that t' xarg ( -3 ) Iïx(x )=x = , since and ,i -1 = ( ï) e u t'= arg [(-]z)()] = arg(). By a flrther application of Lemma 2.7 we obtain I)q(X O) = Vw2 (xarg( 3)) with _ 2F t" t" = arg ['r(wz)( )] = arg [(91 ï)( - )] = arg(-). = x '+g(]), which proves the lemma. Hence, I/), (x °) = .r arg ( ) [] CLASSIFICATION OF IANK ONE CHAIN DOMAINS 2747 In order to construct further cones we consider a, subglotlp H of G that contains F and defilm PH =Hn?. It follows imInediately that PH is closed under multiplication. If g HPH, then g and g- 6 H a = PH follows; PH is acone of H. Lemma 3.3. Let t . Then PHXt = xt. Proof. Itis enough to prove this for t 6 [0, ), since t = k + t0, t0 6 [0, ) in the general case with x in the center of G. If t = 0, then PHXt = = xt. If t 6 (0, ), then for any j 6 J() there exists an n with wîxt j by Lelnlna 3.1. Hen«e, xt = J() = Uwîxt ç PHxt, and the statelnent in the lemlna follows. The next result shows that F contains eleinents of a certain type. Lemma 3.4. For any integer m and ay > 0 there ests an elemet xtu in F with " U ad t (m, + ). Proof. Let h be the element www in F. Thon, by Lemma 3.3, we have I(x +) = x ° where fl = arg(). It follows that lh(x °) < l7(x ) = x - and that tkç (x °) < x -for any natural number N. conclude that for the given integer , there exists a natural mmber N and an integer AI < m with For e the given rem number, there exists by Lelllma . and tlle contimlity of I a with 0 < < e and «, ([°, x)) ç (,x+) and hence (.) , ([x«+)) ç ((+«(+1+) follows for all k Bv Lemma 3.1 there exists a natural number n with çÇï,h(x °) [xM,x M+5) and h=h F. Hence, by (*) we obtain By another application of Lenlllla .1, there exists a lmtural llllnlber 2 with By repeating the last two steps m - (M + 1) tilnes, the statement of the lelnllla follows. The next result shows that the cones P are indeed exceptiolml. Proposition .. Let H F be a sbgrop of ad P = H. The: a): P is an ezceptioal rak oe cone i H. defiees a isomorphism betwee () ad (P). rhe iwverse 4 ç is 9iven b ç-(C) = C for C a divisorial P-ideal. 2748 H. H. BRUNGS AND N. I. DUBROVIN Pro@ We recall that AA(IF) is the group of divisorial IF-ideals in G (Definition 1.6) a.nd that AA(IF) = gr {Q} = gr (xN?) by Theorenls 1.9 and 3.2. If C is a divisorial PH-ideal in H. then C'IF is a IF-ideal in C by Lenmla 3.,-1. The divisorial closure C' of CIF is thereh)re equal to solne power of ,rrIF and C' = follows for some m in ]. çVe want fO prove that CI? ç3 H --: C and assulne that hPH D_ C for some h C H. Then hPHIF = hIF D CIF; hence hIF __D CIP. Therefore. hPH = hPçH D_ CIFç H. It follows tlmt C= C D_ CIFçH D_ C" and C" = CDH. This shows that Cbeing a divisorial PH-ideal ilnplies C = xIF ç3 H for some m. We want to show next that (z'IF-"-H) : :r''IF H for anv u. Since is divisorial, we know that (a«'IF-'H) = xIF (3 H for SOlne m by the above al'glllll(qI. Bv Lemma 3. there exist elelnents x tul, xt2uz F C H with N, u,,u2 G U a.nd ri,t2 G (Tf(tf - 1).Tf(t, - 1)+ ). It fillows that rt'l PH = u.t UlIF f-? H D .r "- u2IF H = x t u2Pu x P H. Hence, .r(n-l)P H (.rPH) xP H. If «(-I)pH = (zPH), then this ideal would also be equal to .rtPH and r t: uPH. This would ilnply xttl PHP = zP = ,rtuPH = ztP, a contradiction that shows that (.r H) = (zP H) for all n. This set of divisorial P-ideals does hot contain J(PH), does hot contain a completely prime ideal (Lelnm 3.3 and 3.5) and no ideal of the folIn zJ(PH) ¢ J(PH), « PH, is colnpletely prime in PH. This shows that PH has rank one and that «(PH) is infinite cyclic with QH = z H as the positive generator of ,M(PH). Since J(PH) D QH. it follows from Theorem 1.9 that PH is a.n exceptional tank one cone in H. This proves all sta.telnents in the lemma. We consider now a condition that will guarantee that PH is exceptional of type (c). Proposition 3.6. Let H be a subgroup o.[G containing F with H (gr {x'} x U) = gr {x 'k} x U(PH) for some integer lr >_ O. Then the exceptional cone PH bas type (C). Pro@ It was shov«n in the previous proposition that PH is an exceptional cone with AA(PH) = gr{(xIF (3 H)}. To prove the statement in this proposition it must be shown that 7-l(PH) = gr {xPH}, sec Theorem 1.9. Hence, let gPH = PHg be a principal ideal in H (sec property e) iii Section 1.1). Then 9IF = PHgIF = IF9 ]P by Lenmm. 3.4 and gIF = IFg since IF has rank one. By Theoreln 3.2, c) it follows that g = .r" G H for some integer m and u c U and /7 = Xkr for u U(PH) and solne integer n bv assumption. Therefore, gPH = xrrrpH and H(PH)= gr {xrtpH} = gr {Qk} follows for Q = xIFçH: PH is exceptional of type (Ck). [] Theorem 3.7. Let Hk = gr {u) 1, W2,.r rrk} be the subgroup ofG generated by F and the central element x for an integer k >_ O. Then Pk = IF V Hk is an exceptional tank one cone in Hk of type (Ck). Pro@ It is sufficient to verifv the conditions iii Proposition 3.7 for H.. CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2749 Assulne that (*) "]Ç/r/';P'/ïl'/-'I /J-'ï2"/-' 2"'" u'ïnw n = .r=m* Hk R (gr {z =} x U) for solne integers p. ui, Pi for i = 1 ..... , and u U. b apply the mapping (Theorem 2.5b)) to both sides of thc above equation and obtain 12Wl 1 î)...(2ïn 1 a 1) (**) (-1)P(o 1 )(2"1 where u = ( 0 _1) with b, 0<aGN. Since the entries of the lnalrices al the left side are all integers, it follows that and a - are int.egers greater lhan Zel'O: hence a. = a - = 1. By a silnilm" argument it follows that b is an even inleger, b = 2s for some s in Z and u = ( ) = ( î ) = wf G r(F) follows. If (-I)P(-1) = -1, then it follows from (**) that __(î) = (2ïl)(2 1 î)...(2n)( 1 ..î)( ) e (F), which is a contradiction, since the group r(F) fl'eely generated bv r(u,1) and (see the relnarks bcforc Lclmna a.3) does hot ('ontain a nonIrivial central elelnCnt. Therefore, (-1) = (-1) can 1,e can«clled in (**) and, using again Ihe fa.ct that b = 2tq if we ignore expon«nls lhaI could l»c zcro. Wilh u = wï ve tan rewrite = z tv . Il fifllows that tq = s, m = kp and u = the condition in Proposition 3.7 is satisfied and Theorem 3.8 follows. 4. EXAMPLES OF EXCEPTIONAL ANK ONE CHAIN DOMAINS In this section we construct domains S associated with the exceptional cones Pk of type (Ck) as described in Theoreln 3.8. In Lemma 2.6(c) it was proved that TU is an Ore domain for any skew field T and the subgroup U of G. denote by K the skew field of quotients of TU for a given skew field T; for exalnple, T = Q, the ratiolmls. Let lç{G} be the right K-vector space and Ieft T-vector space consisting of all series with ri < t < .... k Iç. and supp () = {a "t' [ k ¢ 0} well ordered. XZ call snpp () the support of the series . If k ¢ 0. lhen v() = z ri 6 R is the norm of 7 and v(0) = for 7 = 0. Let Q = End Iç{G}- be the elldolnorl)hisln ring of the K-veclor space Ix {G}ç. For q G Q and G K{G} we write q[7] for the ilnage of under q. The representa- tion V : G Aut (R, 2) considered in Section 2.3 can be extended to a mapping V defined on Q with V(') = '(q[.']), V() = for q Q, .t 6 R, and l (R. OE) (R, oe). It follows that ç+(a "t) E nlin{ç(z'). for any a, b Q and z t R. However, 1' is not equal to I o 1 in general. We recall a definition and a result given by Mathiak in [17]. Definition 4.1. Let D be a skew field and (F, ) a linearly ordered set. Then a lnapping 1" : D* Aut (F, ) is called an M-valuatio if the following conditions hold: r MV1. I = I o l,g for any a,b D*; 2750 H. H. BRUNGS AND N. I. DUBROVIN MV2. Va+b(h) >_ mill{l/(]), I/b(]))} for any a,b D* with a + b -)/= 0 and h C. If we add the syInbol oc for infinity to F and define V0(b) = oc and t(ec) = oc for all h F, 0, a D, then MV1 and MV2 will be valid for all eleInents a, b D and all b F U {et}. The next result follows almost directly froIn t.he previous definition; see also [16] nd [17]. Proposition 4.2. Let V D* Aut(F, _<) be an M-valuation for a skew field D and a linearly ordered set (F, <_) and let 5 be an element in F. Then the set Sh = {d D I Vd(h) _> 5} is a total subring of D. Conversely, any total subring S in a skew fleld D can be obtained in this way for F = {aS I a D* }, aS >_ bS if and only if aS c_ bS and I)(aS) = daS. The ring S coincides witb Sh forh=S F. [] The space K{G} introduced above is also a left G-module if we define for g G and 3' = '. xtlki Iç{G} that g3` = :C t' (Ulkl) n t- xt; (U22) n t- xt (u3k3) -t- . where g. x t = xt',i for ui U ç K, t' i ll. It follows from Lemma 2.6(d) that t' < t < t < ... is also well ordered and hence g7 K{C}. The group ring TC can therefore be considered as a subring of Q. If A is any subring of Q. then we define 79[0. ,4] = A and 79[n + 1, A] as the subring of Q generated by 79[n, AI and ail inverses of elements of 79[n, ,4] in Q. The union U 79[n, A] = 79[A1 n=0 is called the rational closure of A in Q. Let I13 = 79[TG] be the rational closure of the group ring TG in Q. The following result can be round in [11] (see [10] also): Theorem 4.3. a) The rational closure I of TC in Q is a skew field. b) The mapping 1 restricted fo I* is an M-valuation of I* to Aut (R, _<). c) The rang S = {d I13 I Vd(x°) x°} is an ex«eptional tank one chain order in of type ( C ) associated with the exceptional cone P in the group G. In order to construct skew fields that contain rank one exceptional chain orders of type (Ck) we consider the rational closure Dk = [THk] of the group ring TH for the group H = gr {u,,w2,x "} (see Theorem 3.8) in Q = End K{G}3. Since Dk ç = [TG] C Q and is a skew field by the above theorem, it follows that D is also a skew field and S = S D is a total subring of D. It follows from Corollary 1.10 and Theorem 3.8 that S is an exceptional rank one chain domain of type (Ck) if the following theorem is proved: Theorem 4.4. The total subring Sa = S Dr is associated with the cone Pk = P H. Before this theorem can be proved, we need the result in the following lemma. Lemma 4.5. Let K{}. Then (,) suppd[? l ç I(supp?). dD gH CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2751 Pro@ Let I: be the right side in (,). Then iii order to prove (*) it is sufficient to prove (**) supp d[T] C_ }: for any 7 K{G} and any d Dk = U [n, THk]. X will prove this in rive st.eps using induction on n for the smallest index with d [n, THk]. STEP 1. Assmne that d = xtu ¢ Hk, u ¢ U and that i<A 0 ki K for all ordinals i < A. Then d = xt:(uiki) for .rtu.r t" = xte,i , ai ¢ U. Hence, suppd[] = i<A suppd= {x t{ l i< A} = {lh(xt') [ i < A} Ç {I(xt') l g¢ H.,i STEP . The inclusion (**) follows ilnlnediatelv for d ¢ T. STEP 3. Assulne that o, b Dk with suppa[] U suppb[] ç I for any ¢ Iç{G}. Then supp (a+b)[] ç supp a[]U suppb[] ç }ç. Further, I(}') = } for g ¢ Hk and hence "PP( )[q ç ;[ = U ç("pp[q) ç U ç)= ,. gEHk gEHk STEP . It follows froln Stcps 1-3 that the statement (**) is true for ail d STEP 5. Assume that (**) is true for elements d [, - 1, THk] for some n 1 and all 7 K{G}. Let d = p- [,,. THk] with p [, - 1, THé.]. XX consider fl = d[7 ] and decompose fl into the sure fl = flo +fl with supp (flo) Th, v = p[Z] = p[£] + By the induction hypothesis, it follows that gHk gHk Hence, supp (pitié]) = supp (7-P[o]) ç supp TU supp (pi&)) ç ]. On the other hand, supp (pitié]) ç ]' since p e [n - l, THk]. If we assume that there exists element h in supp (pitié]). then, on thc one hand, forsome gHk andsome h' supp(7) h = Ih' and on the other hand. = V« (h") for solne g' E Ha. and solne h" E supp (fll)- This ilnplies h" = I@,}«9(h' ) Yv ç supp (/31) = 0, a contradiction that shows that supp (fl) is empty and supp (ff) = supp (flo) The ring [n, TH»] is generated by [n - 1,THk] and all elements p- for p [n - 1, THk]{O}, and it now follows bv an application of Step 3 that (**) is true for ail elements in [n, THk] which completes the induction and proves the lemma (see also: [111). We now return to the proof of Theorem 4.4. Let d be a nonzero elenmnt in D. Since Dk ç D and S is associated with the cone , the element d can be decomposed as follows: d = xtm = qx t' with m,q U(S), xt = xt' 2752 H. H. BRUNGS AND N. I. DUBROVIN (see Definition 1.2). It, follows from (*) iii Lelmlm 4.5 with 3' = x° that suppd[x °]C U I('r°)= U v(g}. gEHk gEH Hence, ,(d[x°]) = l(.r °) = I( o l(x °) = l:,(.r °) = ,c t since m U(S), and hence .r t = v(g), g = ,rv, « U for SOnle elelnent g H. It follows that d = (.rtu)(u.-m) for xtu H and u-m = (xtu)-d Dk. Further, u-m U- U(S) Dt. = U(St.), since U Ç U(S) and S Dt- = S. Applying the salue argulnent.s to the elenlent d - = x-t'q -1, we conclude that there exist.s an elelnellt f Hk with 9 = x -t' w for SOlne w U. Hence, we obtain a decolnposit Joli d-t t' lq-1) -lq-1 = (x- u,)(u,- for « = (g,)-ld- e Dt. a lz(ç) = U(S). This proves the first hall of condition (ii) iii Definition 1.2. if we write d = (qu,)(u,-.rt'), q«, U(Sk), w-ix t' = (9') -i H. It renlains to prove the equality Px,P = pu-.r'p.. Let w-ix t' = ,t".t for sonw u U and t N. Since q" is associated with , it folh)ws that Therefore, IP.rt'lp = Ipxt"Ip. P.rtuP = Pt.xtu(Y c H) = Pt.xtY t Hk = lPxtlp çl Hk = IPxt'IP ç Hk = II%t"IP ç Hk [] Corollary 4.6. Tbe cbain domain Sk is exceptional of tank one and of type (Ck). 1:1 EFERENCES 1. G. M. Bergman. Right-orderable groups that are not locally indicable. Pacific J. Math. 147 (1991), 243-248. MIR 92e:20030 2. H. H. Brungs, H. Marubayashi, and È. Osmanagic. A classification of prime segments in simple Artinian rings. Proc. Amer. IvIath. Soc. 128 (2000), 3167-3175. M1R 2001b:16055 3. H. H. Brungs and M. Schr6der. Prime segments of skew fields, Çanad. J. Math. 47 (1995), 1148-1167. MR 97c:16021 4. H. H. Brungs and G. T6rner. Chain rings and prime ideals. Arch. Math. 27 (1976), 253-260. M1R 54:7537 5. H. H. Brungs and G. T6rner. Extensions of chain rings. Math. Zeit. 185 (1984), 93-104. MI 85d:16012 6. H. H. Brungs and G. T6rner. Ideal theory of right cones and associated rings. J. Algebra 210 (1998). 145-164. MR 99k:20113 7. P. M. Cohn, Skew Fields, Çambridge University Press, 1995. M1R 97d:12003 8. N. I. Dubrovin. Chain domains. Moscow Univ. Math. Bull. Ser. 1. 37(1980). 51-54. 81g:16004 9. N. l. Dubrovin. An example of a chain prime ring with nilpotent elements. Mat. Sbornik 48 (1984), 437-444. M1R 84f:16012 10. N. I. Dubrovin. The rational closure of group rzngs of left-orderable groups. Mat. Sbornik 184(7) (1993), 3-48. MR 94g:16035 CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2753 11. N. I. Dubrovin. The rational closure of group rings of left-ordered groups. SM-DU-254, Duis- burg, 1994, 96 pp. 12. T. V. Dubrovin and N. I. Dubrovin. Cones zn groups. Sbornik Mathematics 187(7) 1996, 1005-1019. hIR 98c:20082 13. L. Fuchs. Teilweise geordnete algebrazsche Strukturen. Vandenhoeck und Ruprecht, G6ttingen, 1966. hIR 34:4386 14. V. K. Goel and S. Iç. Jain. n-injective modules and rings whose cychcs are r-injective. Comm. Algebra 6 (1978), 59-73. hIR 58:11016 15. bi. I. Kargapoiov and Ju. I. Merzljakov. Fundamentals of the Theory of Groups. Springer- \rlag, 1979. MR 80k:20002 16. N. lklathiak. Zut Bewertungstheorie nicht kommutativer K6rper, J. Algebra 73, (1981), No. 2, 56-600. MR 83c:12026 17. K. Mathiak. Valuations of skew fields and projective Hjelmslev spaces. Lecture Notes in hlath. 1175, Springer-Verlag, 1986. MR 87g:16002 18. B. L. Osofsky. Noncommutative rings whose cyclic modules bave cyclic inective hulls. Pacific J. Math. 25 (1968), 331-340. lkIR 38:186 19. E. C. Posner. Left valuatzon rings and simple radical rings. Trans. Amer. Math. Soc. 107 (1963), 458-465. hIR 27:3665 DEPARTMENT OF IIATHEMATICAL AND STATISTICAL SCIENCES, UNIVERSIT , OF ALBERTA, ED- MONTON T6G 2G1. CaNaDa E-mail address: hbrungs@math.uaXberta, ca DEPARTMENT OF IIATHEMATICS, VLADIMIR STATE UNIVERSITh. VLADIMIR, RUSSIA E-mail address : ndubrovin@mail.ru GoR STR. 87. 600026 TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2755-2769 S 0002-9947(03)03270-7 Article e|ectronically published on February 25, 2003 ON THE SPECTRAL SEQUENCE CONSTRUCTORS OF GUICHARDET AND STEFAN DONALD W. BARNES ABSTRACT. The concept of a spectral sequence constructor is generalised to Hopf Galois extensions. The spectral sequence constructions that are given by Guichardet for crossed product algebras are also generalised and shown to provide examples. It is shown that ail spectral sequence constructors for Hopf Galois extensions construct the saine spectral sequence. 1. INTRODUCTION A. Guichardet [3] has given two constructiols for a spectral sequence with/ïT 'q : HP(G, Hq(], X)) and target the Hochschikl cohonlology H°(G x« B, X) of the crossed product algebra A = (7 x B. where (7 is a group, B is an algebra, c is a represeutation of (7 by automorphisns of B and the nmltiplication in A is given bv (g,b)(g',b') = (gg',a-,l(b)b'). Here, X is a left and right A-bimodule or, equivalently, a left module over A « = A ç-) A °p where A °p is the opposed algebra of A. These constructions are mmlogous to the Hochschild-Serre constructions for the spectral sequence of a group extension. He has asked if the methods of Barnes [1] can be used to show that they construct the smne spectral sequence. D. Stefal [5] has given a spectral sequence, based on the Grothendiek composite functor spectral sequence, for the cohomology of a Hopf Galois extension. The three contexts have some features in common. All have a "large" algebra A and the category .A of A-(bi)modules, a subalgebra B, a "small'" algebra (7 which plavs the role of a quotient, of .4 by B and the cat.egory C of Cnlodules and a category/7 in which the filtered cochain complexes are constructed. Ail have a left exact flnct.or ¢ .A --, C and a left exact functor ¢ C --, /7, raid the spectral sequences have as t.arget the right derived functors of the composite 0 = ¢ o ¢. Throughout this paper, ¢, ¢ and 0 will denote these functors. In Barnes [1], A is an auglnented algebra over a commutative ring .q and C is the quotient AffB of A bv a normal augmented subalgebra. A, B and A/lB are all assumed to be projective as .q-modules. The functor ¢ is given for the left A-module Received by the editors April 30, 2001. 2000 Mathematics Subject Classification. Primary 18G40, 16W30; Secondary 16E40. Key words and phrases. Spectral sequence, crossed product, comodule algebra, Hopf Galois extension. This work was done while the author was an Honorary Associate of the School of Mathematics and Statistics, University of Sydney. ()2003 American Mathematical Society 2756 DONALD W'. BARNES X by eX-- XB= {.cE .\'lbx=e(b).r fo," all bE/3}. For a C-module }', P} = }-c and we have ppx = X A for any A-module X. In this context, p has a left adjoint j C - al, and use is ruade of the counit 7r = jp al - al of the adjunction. Note that pj = id C - C. \Ve shall refer te this as the HS context. In Guichardet's paper, A is the crossed product G x«/3. where/3 is an algebra over the field .ff, C is the group algebra JïG and the fimctor p is given for the left A«-module X by OX = X B = {.'r E X I b.r = :rb for all b E B} with the action of G given by g.r = (g, 1)x(g -1, 1) for g E G and x E X B. For the .¢G-module e"}" = }-G _ {y E }'IgY = Y for all g E G}. In this context, çb does net bave a lefl a(tj()int j. The assumption that .ff is a field can be weakened if in seine places we tel)lace "injective'" by "relativelv injective". \Ve require that . is a c(mmmtative ring and that B is )ï-projective. \Ve shall refer te this as the G context. In Stefan [5], C is a H()i)f algebra over the field ., A is a C-comodule algebra and B is the subalgebra of coinvariants. Thus we have an algebra morphisln XA " A -- A C making 4 a right C-comodule, and B---Aç°c= {«,E A I,X4()---® Stefan refers te this situation as "'the extension ,4/13"'. It is assumed te be C-Galois. which we explain in section 2 below. As in the G context, for the A-bimodule X, we set px = X B. This is ruade into a right C-module using the action defined by Stefan [5, Proposition 2.3] and explained following Lemma 2.1 below. For the C-module Y, we put ¢;'1" = }-c. Again, we veaken the &sumption that .ff is a field. \Ve require that . is a commutative ring and that A. B and C are )ï-projective. We refer te this as the S context. It generalises the G context since the crossed product algebra A = G × « B becomes a C-comodule algebra if we set x.4 (9, b) = (9, b) ® 9 for 9 E G and b E/3. If in the HS (Hochschild-Serre) context, .4 is a Hopf algebra. we mav regard it as a C-comodule algebra with the comodule strncture given bv the conmltiplication of A followed by the natmal homomorphism .4 ® 4 -- .4 ® AffB. A left A-module X mav be regarded as a 1)imodule by setting xa = e()x for a E _4 and x E X, where e is the augmentation. This does net change çX = X , new defined as {x E X ]bx = xb for all b E B}, ner does it change the Rqcp(X), although it does change the injective modules used for their calculation. 2. PRELIMINARIES We follow the notation for comodule algebras used in Schneider [4], with the exception that we denote the Hopf algebra by C, reserving the symbol H for coho- mology. Thus we have the comodule structure map, AA : A --, A ® C and express the image of an element a E A bv A.«(a) = a0 ® al. The comultiplication Ac : C C0,C is written Ac(c) = yc ®c2. The augmentation of C is denoted by e and the antipode by S. The canonical map ean : A ® A ---* A ® C is defined by can(a ®B a') = aao ® al . That A/B is a Hopf Galois extension means that ean is invertible, which we always assume. Thus for c E C. there exist elements SPECTIAL SEQUENCE CONSTIUCTOFIS 2757 ri (c), li (c) E A, hot mfi<luely detern,ined, such that tan- 1(1 ®c) = ri (c) ®B li (c), which is unique. \Ve shall need thc identities proved in Schneider [4. Ptemark 3.4(2)]. Throughout his paper, Schneider assumes .ù, to be a field, hut his proof of the iden- tities makes no use of that assumption. For the convenience of the reader, we list the identities here. Lemnla 2.1. For all a .4. b t3 atd c, c' C, t]e followin 9 identifies ]old: (a) bri(«) ®B li(«) = ri(c) B li(«)b, (b) () («)h(«) = («). (d) r,(«).i(«)oli(«) = ,',(c)li(c)c2. () () Following Stefan [. Proposition 2.], we use the above relations to d«fine a right Cmodule structm-e on X we put .r- c = ri(c)xli(c). This is well defined since ri(c) B l(c) is a well- defined elelnent of A @B A alld b.r = .rb for ail b B. Froln 2.1(a), it follows that x. c X B. If a left açtion of Ç on X B iS preferred, o11(' lllaV be defined by setting c- = x-(Se). The sertion of LOlnlna 2.2 below holds for this lef action provided that the antipode S is bijective. Lemma 2.2. For an Aç-modle X . ( X e) c = X A. Proof. For z X A and c C, we bave by 2.1(c). Thus z (XB) c. Conversely. if x (XB) c, then by 2.1(b), for all For the crossed product algebra A = G x B, the canonical lnap is given bv b Cl((,)('.v'))=(, )(,') =('.@()'). In particular, can((9 -1, 1)B (. 1)) = (1.1): so we can take r(9) = (-. 1) and l(9) = (9.1)- The right a.«tion of C G on X B be«onmsz-9 = (9-1.1)z(9.1) for z X and 9 G. Converting this to a left action gives z = (, 1)z(9 -1. 1). which is the action used in the Guichardet paper [3]. Note that, in the G context,..4 « = A A °p is free as a right B«-module. In the S context, we assunle that A is fiat as left and right B-module. If tlmn follows that A « is fiat right B«-lnodule. In the HS context, we assume that A is at least projective as right B-module. At some points in [1], the stronger assunlption that the modle quotient A/B is projective as right B-module is use& In all the contexts, by Barnes [1. Lelnlna 1.4.3], everv il0ective left A- or 4C-module is injective as B- or B«-module. Further, every il0ective of M or C is injective as .q-module. For any .q-module X, the A-module coinduced froln X is the module X* = Hom.(A, X) wit.h the action (af)(a') = f(a'a) for f X* and a,a' A. (For the coinduced right lnodule, the action is given by (fa)(a') = f(aa').) If X is itself a left A-module, then the nmp « X X* defined by («.r)(a) = ax for a" q X and 2758 DONALD W. BARNES a E A is a .R-si»lit A-module monomorphism. X* is a relatively injective left A- module. (See for example, Barnes [1, Lenuna II.2.4, p. 20].) A module is relatively injective if and only if it is a direct sunmmnd of a coinduced module. By Barnes [1, Lenmm. II.3.9, p, 28]. if A is right B-projective, then everv relatively injective left A-module is relativelv injective left B-module. Thus in the G context, every relatively injective A«-module is relatively injective as B«-module. In the S context, to get this conclusion, we nmst strengthen Stefan's assumption that A is left and right B-fiat t.o A being left and right B-projective, although as t.he next lemma shows, this strengthening is mmecessary if, as in [5], if is sumed that is a field, since then, every module is -injective. Lemma 2.3. If X is -injeclive, then the coinduced module X* is injecive. Proo: Let i I" IV be a monomorphism and a I" X* a honomorphism of A-modules. i X* We want to construct a. homomorphism fl II" X* such that i = a. For v I , we bave (tv Hom.(A, X); so for a A. we lmve (v)(a) X. m- regard a as afimction A x 1" X, writing a(a,v) for (av)(a). For b A, we have a(bv) = b(av). So (a(bv))(a) = (av)(b); that is, (ab, v) = a(a, bv), and in particular (a. v) = a(1, av). Putting (v) = (at,)(1) creates the diagram of -modules. Since X is -injective» there exists a -homomorphism I1 X such that Bi = . Define " II" X* by (w)(a) = B(aw). Then for b e A. we have /(b)() = (b) = (,,,)(b) = ((,,,))(); so/3 is an A-module homomorphisn. have (,,)() = $((i,,)) = ((,)) = (,,) = (,)() and fil = a. We have defined a right C-module tion on X u for any leh A«-module X. need another description of that action in the case where X is a coinduced module. Lemma 2.4. Let V be a R-module and let I'* = Hom(A , V) be the coinduced A - module. Then V *B is isomorphic to the coinduced C-module Hom(C. Hom(A, 1")). Proof. The tion of a @ a' A « on f V* is given by (( e ')I)( e )= I(( e )( e '))= for x, y A. Now Y*= {f Y* I (be )I = ( e b)I for fi } = {f V* f(xb@y) = f(a" @ bç) for ail b B and x,y in A}. SPECTRAL SEQUENCE CONSTRUCTORS 2759 Thus 1"* B can be identified with Hom(A®u A, V), and so, using the canoifical map, with Hoin(A ¢_ C, 1;) = Hom(C, Hom(A, V)). We calculate the right C-modnle action Oll Hom(C, Hom(A, V)) induced by these identifications. For f V *B and c G C, froln the A%a.çtion on 1", we get fc = 7,(ri(c) C' li(c))f. Thus (fc)(x ® g) = f(xr,(c) ® li(c)g). For f Hom(C, Hom(A, V)) we have the corresponding f' Hom(A ® C, 1") given by f'(o ®c)= f(c)(a) and f" Hom(A ®8 A. V) givell by f"(x ®B U)= f'(ean(x ®B y))---- f'(xgo ® U)- Thus, f(c)(a) = f'(o®c)= Zf"(ori(c)®Bli(c)). So for c' C, fd is given bv (fc')(«)(a) = Z(f"«')(ari(«) ®B li(c)) = f"(""(«)"(ç') ®- = f"(ari(c'c)@.li(c'c)) by 2.1(f) = f(d«)(a). Thus the action (fc')(c) = f(c'c) is that of the coinduced right C-module. [] The next result strengthens Stefan [5. Proposition 3.2]. The corresponding result in the HS context follows easily froln the fact that every AffB-module is an .4- module and that QB is a submodule of Q. Lemma 2.5 . Let Q be a relatively injective A¢-module. Then QB is a relatively injective C-module. If Q is injective, the QB is injective. Pro@ Q is a direct smnmand of some coinduced module I "* = Hom.(A , 1"). So to prove Q8 relatively injective, it is sufficient to show that Hom.(A «, V) B is relatively injective. But Hom(C. Hom(A. 1")) is relatively injective; so by Lemma 2.4, V *B is relatively injective. Thus QB is relatively injective. If Q is injective, we nav take V = Q. Since Q is .¢binjective, by Lemnm 2.3, Holn(A. Q) is also .R-injective and Hom(C, Hom(A, Q)) is an injective C-module. Thus QB is injective. [] 3. SPECTRAL SEQUENCE CONSTRUCTORS GENERALISED In Barnes [1, Chapter III] in the HS context, a spectral sequence constructor for (¢, ) was defined to be a functor F from ,4 to filtered cochain COlnplexes in T) such that (1) F is exact (in every filtration). (2) F is acyclic on injectives; that is, if Q is injective, then H'(FQ) = 0 for n > 0 and for all p. Hq(°FPQ/°Fp+IQ) = 0 for q > 0. (3) E°F is exact on C. (4) The inclusion i" X B --, X induces isomorphislns Eï°F(X B) --, Eï°F(X) for all p and ail X .4. (5) H°F is naturally isomorphic to 2760 DONALD W. BARNES Here, using the fact that A//B-nlodules are A-modules, that is, using the adjoint j to b, we c'onstruc't a flmc'tor F : '" 1 fi'Olll Cto cochain complexes in . This cannot be done in the G or S contexts. So we nmst include the cochain complex flmctor as part of the structure in our definition of a constructor. If F is a filtered cochain complex, we denote the colnponent of total degree n bv F, the pth filtration bv F', the submodule of filtration degree p and C'olnptementary degree q by F pq and use similar notation fbr the t.erlns of its spectral sequence E(F). e always assmne that "F ° : "F and that 'F +1 : 0. The following definition generalises the Olle quoted above. Definition 3.1. A spectral seqllcnce constructor for the pair (, ) is a quadruple (F. F. /. 7), wherc F is a functor from A to filtered cochain complexes in , F is a fimctor ffoto C to cochail c'omplexes iii , I I is a natural isonlorphisln l ("' F) F& and 7 is a natural isomorphism H°(F) ç such tha.t (1) F is exact (in everv filtration). (2) F is acyclic on injcctivcs: that is, if Q is injeçtive, then H"(FQ) = 0 for n > () a.nd tr ail p, Hq('FPQ/"FV+Q) = () for q > 0. (3) I" is exact an(t a.«y«lic on inje«(ives. From (1), il follows that "FV'F + is exact for all p, r. From 7 being a natural isomorphism and (3), it follows that we have a torique fanfilv of natural isomor- phisms 7 p Ht'P + RPç, the right d('rived flm«tors of , «ommuting with colnmct- ing homomorphisms, and with 70 = 7- k, denote this family bv 7. Furthermore, H'F = E°°F._ = E°F = H°Eï)F = H°q-iF, and SO "H°(71) H°F = 7 = O: that is, "7H°(7/) is a lmtural isonorphism flOlll H°F to 0. It follows that H F = RO for all n. X now prove thc results corresponding to Barnes [1. Theorem III.2.3. p. 42] and the lelnm leading up to that theorem, begimfing with the analogue of [1. Theorem III.l.5, p. 38]. Lemma 3.2. Let (F, F,I.) be a spectral seq,ence con.tructof There exists a unique famil of natw,'al t','ansformations ,1 pq " Eï ( F) F" ( H q¢) coin m uting with connecti9 homomorphi.s'ms and with I1 pO = I1 p. Ile ]Pq are atural isomorphisms. .sati.sg I]P+l"qdï q : (-1)qgl] pq. where d is the differential 4 F. and induce isomophisms (,'")- () and HP(q°q) - Eq(F) --> (R'b)(Rq¢). Pro@ F is exact by assumption. So {F"(Rq¢)[q = 0.1 .... } is a connected sequence of fun«tors, as is {Eï(F) = Hq(F"/F"+)]q = 0.1 .... }. Both vanish for q > 0 on injectives. By dimension-shifting, it foltows that there exists a torique family of natural transformations 1 "q - Eï(F) ['P(/ïqçD) extending the given tra.nsfor- mation qP Eï°(F) ---, FP¢. Since /p° is a natural isomorphism, all the t' are natural isomorphisms. The argument of [1, pp. 38. 39] applies unchanged to give the result. [] SPECTRAL SEQUENCE CONSTRUCTORS 2761 Definition 3.3. A natm'al transibrmation ( (F. F. q, 7) -- (F'. F'. q'7') of con- structors is a pair F " F -- F and çFÇ -- I , Of natural transformations such that the diagralns COlllnlllt (L VVe shall onfit thc subscripts from çF and çr- Out next lemma is easier than [1, II.2.2] in that the transformation ç F -- F' is given instead of having to be constructed. Lennna 3.4. constructors. Let (F, F, '1, 7) -- (F', F'. 1', 7') be a natural tra,fformatton of Then the diagram :ït ( r ) -- FP Rqc Eïq(F,) '/ , F'PRqdp commutes for all p and q. Pro@ çH°ç is a natural transformation of connected Se(lUellCeS of fimctors. Since in dimension q = 0, we have R°¢ = I//P0(I/P0) -1, tV dimension-shifting, we have çq¢ = ?]tpqÇ(?]pq)-i for ail q. Theoreln 3.5. Let ç (F. F. q, 7) ( F. F, q', 7 ) be a natural traufformotio, of constructors. Then ( induces a natural isotorphism. çE " E(F) E(F ) of their spectral sequences, that is, (.q Eq(F) Eq(F ) is a natural isomorphism for all r 2 and all p. q. Proqf. Bv assumption, " H°(r) , and 7' " H°(F') are natural isomor- phisms, and 'H°(ç) = . Therefore H°(ç) = (')- is a natural isomorphism. Bv dimension-shifting, it follows that HP() " H H is a natm'al isomorphisln for ail p. The diagram is, up to sign, a commutative diagram of cochain COlnplexes. (If q is odd. q and q' anticommute with the differentials.) Taking H p of this, ve get the commutative diagram in which '1, '1' and HP(ç) are natural isomorphisms. It follows that çq is a natural isomorphism and so, that ,,cm is a narreal isomorphism for all r _> _.° [] 2ï62 DONALD "vV. 13ARNES GI. ICHARDET S FIRST ('ONSTRUCTOR Foi" the Ae-lnodule X, Guichardet defines the double colnI)lex içpq(X ) = (2r(av+I,Hom(®V+IA«,X))B)a with apI)ropriately defined differential, where .T-(b, V) denotes the set of furie- tions rioto the set U to the set I . Following Guichardet [3], we set I'(X) = Hom(® '+IA , X) with differential ! d,f(oo ® o'o ..... o,+, ) (/ri+l) ! = a0,f(o ®(4 .... ,o+ ( ' , , + -- 1)i+,f(ao ® o' o ..... aiOi+l @ ai+lai,., an+l @ an+ 1 0 which gives a relatively injective resolmion l'(X) of X in A. Also following GuMmrdet, we put Pn = ®'+I-R(,' with action g(9o ®... ® .On) = .O.Oo ® ... ® .O.On and set d(go ®... {" g,,) ' ' = = Yi=0(-1) (9o®...'i. -®g,,) and e(90) 1. This makes Po a ff'ce resolution of . iii C. We then have lçoq(X) = tlom.G(Po, Iq(X)B). Expressed iii this way, it is the Grothcndiek repeated (relatively) injective resolution construction foi the spectral sequcnce of a composite flmctor discussed in Barnes [1, Chapter VIII, with Hom.(Po, ) used as the relatively injective resolution fimctor on C. For any relatively injective resolution flmctor I" and any projective resolution P.. setting KPq(X) = Hom.c;(Pp, Iq(X)B) gives a constructor (K.F, 0,7) with F = Hom.c;(Pv, ), r = id and 7 = id. The spectral sequence constructed is independent (from the E2-1evel onward) of the choice of I ° and of P.. This constru«tor mav also be regarded as a.n adaptation of the Cartan and Eilen- berg pair of resolutions constructor dis«ussed in Barnes [1, Chapter VI]. Since 6"7- modules are hot A«-modules, we cmmot use Homa, (Pv, Iq) as in the HS context. but use instead Homc(Pp, (Iq) B) which, iii the HS context, is essentially the smne, Stefan in [5] establishes the conditions foi" the Grothendiek composite flmctor spectral sequence. To obtain a. spectral sequence constructor, we bave merelv to assign ffmctorially the resolutions used iii the construction. If we assume that A is left and right B-projective or if we assume that Jï is a field, then we can use the I n defined as above and an,v right C-module projective lesolution P. of 5. GUICHARDET'S SECOND CONSTRUCTOR For his second construction. Guichardet defines a filtration on the normalised standard COlnplex °N(A. X) where N(A, X) is the subspace of Hom.n(®A. X) of functions f for which f(a ..... a) = 0 if anv of the ai is in .1, and df(al,. .. ,an+l) Yt = al,f(a2 ..... an+l) n u -(--1)i,f(al ..... aiai+l ..... an+l) i=1 + (- 1)'+1 f(al ..... (-In)On+ 1 . SPECTRAL SEQUENCE CONSTIRUCTORS 2763 The filtration on this colnplex is given by defining n_N° tobe the subset of those fllnctions f satisfying f(al .... , aq, glbt,..., gpb v) --i (bl)O, L1...gp(b2 ) o,;pi(bp_i)bp. = f(o1 ..... aq, gl,''', gp)OZg2g3...g p "'" Gnichardet t.akes for F the normalised standard complex and constructs a natnral transformation from E°(N) t.o ['b which, in [3, Lennne 3.11], he shows is a nat- ural isomorphism. The conditions for a spectral sequence constructor are clearly satisfied. The purpose of this section is to gcneralise this to the S context. To nse the nornlalised standard COlnplex in the S context, we must ilnpose a further condition on the algebra A. The theory of the norlnalised standard coin- plex (Cartan and Eilenberg [2. p. 176]) reqnires that the quotient A = 4/.ql be projective as .-module. We assnlne this in this section. Equivalently, we assume that there exists a .-linear map " A --, .q snch that (ki) = k. This condition alwa.vs holds if . is a field or if, as in the HS context. A is ail anglnented algebra. An equivalent definition of nNP, also given by Guichardet, is memfingful iii the S context. So we use it hele but with sides revel'sed because of onr nse of right COlnodnle algebras and right C-nlodnles. Fol" p >_ 1, we define A "p to be the subset of 'N(A, X) of those filnetiolls satisfying (5.1) f(bal,. ..,a_i,an) = bf(al,....a) and (5.2) f(al,...,ai-ib, ai,...,an) =f(al,....ai-l,bOi,...,an) lori=2 ..... p for all a ..... a .4 and b B. The normaliscd standard complex °N(C, )') for a right Cmodule t" is that ob- tained by treating )" as a bimodule with left action c- g = (c)9. Thus, N(C is the subspace of Hom.(®*C )') of functions f for which f(c ..... c) = 0 if any of the ci is in .fil. with the differential n df(cl,.. . ,c+) = e(c)f(c2 ..... c+_)+ Z(-1)if(«l,...,«iCi+l,... ,c,+) i=1 + (--l)n+lf(cl ..... Ch)Ch+l. We put T pq = PN(C, q]v(13, X)) and write N pq for P+qN p. Note that, although N(B. X) is hot, in general, a Cmodule. this does define .-lnodules T pq. For f Nvq(A, X), we put qPq(f)(Cl ..... Cp)(bl,..., bq) = Zri(cP)...ri(c)f(li(c),...,l,(cp),b,...,bq). That O2Pq N pq -- T pq is a well-defined .-linear map follows from the next lelnlna. We shorten the notation by writing (ï for a string al,. , Oq of elements of A of any length. Ve flirther shorten notation by onfitting unnecesary subscripts from the (c),h(c). Lemma 5.3. If NPq(A. X), then for j = 1 .... ,p. and ail A. b t3 and C, C1,...,C p C (1) r(cj) . . . r(cl)f(l(Cl), .... l(cj),) is independent of the choice of the r(cj) and l(c). 2764 DONALD ç\. BARNES (2) '. b'r(cj)'r(cj_l)...'r(cl)f(l(Cl) ..... = r(cj)...r(cl)f(l(c) ..... l(cj)b. 6). (3) y r(c)'r(c)...r(c)f(l(ci) ..... l(cy)l(c).) - r(cjc)...r(cl)f(l(c) ..... l(cjc).ç). Proof. For o. o' ,4. we put gj(a @ a') = a,'(cj_)...r(q)f(l(c) ..... l(cj_),a',g). Bv the condition (5.1), g is well defined. Thus (1) holds forj = 1. Also. by putting , -}B a'= br(Cl) eB /(Cl) = r(Cl) @B bv Lelmua 2.1(a). we see that (2) holds folj = 1. use induction over j. For I <jp, wehave = or(cj_)...r(Cl)f(l(c,) ..... l(c_),bo'.6) 1)v the induction hyl)othesis that (2) holds for j - 1 and condition (5.2). Thus gj is well defined. Putting « '}B o' = r(cj)$ l(c) gives the assertion (1). Putting a OEB a' = b,'(c)@B I(cj) and using Lemlna 2.1(a) gives (2). Putting a 'B a'= r(c)r(cj)@B l(cj)l(c) and using Lelmna 2.1(0 gives (3). Lemma 5.4. defines a nalural cochain map o'" " E'" T v'. Pw@ If f nNp+I, then ¢çq(f)(c,....cp)(b ..... bq) = r(cp) ...r(cl)f(l(Cl) ..... l(%)b, 1,b ..... bq) = O. Since "ç0 = NP/"N p+, Pq defines a .q-linea.r lnap ¢q Eg q T pq. Consider the eXplessiol for (daf)(l(q) .... , l(cp), b ..... bq+). For those terlns in which the string of/(ci)'s is reduced in length, we get bi in the pth place: so those terres are 0. Thus. ( P'q+ l d 4f )(c1 ..... Cp )( bl ..... bq+l) = (-1F ,'(%)...r(q)f(l(c),....l(%)b ..... b+) +(-l:+>(«)...y( .... (Vb+) .... ) + (-:++ («)---/(I(«,) ..... )+ = (-1)PdB((Puf)(q,...,cp))(bl ..... bq+) by applying Lelnlna 5.3(2) to the first terln. The result follows, the naturalitv being obvious. are trying to construct a spectral sequence coustructor using Aç4 = N(A. ) with the Guichardet filtration as the filtered complex functor. Clearly, we tan set F = N(C, ) and 7 = id H°(C, ) . still need a natural isolnorphisln q E°(A4) Y. Applyilg H q to the natural cochain map ¢" gives a natural map qPq Eï PN(C. Hq(B. )) = FP(Hq(B. )). nnlst fil'St show that is al isomorphisln of cochain complexes. SPECTRAL SEQUENCE CONSTRUCTORS 2765 Lemma 5.5. ,l "q " Eï(X) --, F°X B iS a map of cochain complexes. Pwof. An elelnelt of Eï°(.¥) is rel,l'eselte(l 1,y a fllll('tiOll f PNP(A. X) such that f P+INp+I. For f PN p, every terln t in df satisfies (5.1) and (5.2) for ail i except the terre t(a ..... ap+) = f(al,..., ap)ap+l [or whi«h (5.2) lnay rail for i = p + 1. Thus the l'eqlfirPlnelit that dr p+INp+I ilnpOses the OlW extra conditiol that f(al ..... apb)ap+ = f(a .... ,ap)bap+, that is, f(a ..... apb) = f(a ..... ap)b. For su«h an f. we have, wriling ri)l" c 1 ..... Cp+l, (P+'°dAf)( = r(cp+)...r(c)«f(l(Cl) ..... l(Cp+)) = r(«+l).., r(«l)t(«)f(t(«) ..... + (-)'r(«»+)... ,'(«l)f(tt«) ..... («)t(«+l) ..... + By Lemma 2.I(c), = (Cl) r(%+l)...'(«)f(l(ç) ..... By Lcnmm 5.3(3), (-1)(ç+l)... («)/(t(«),..., («,)t(ç+l) ..... (ç+l)) = (-)',.(«+)...,.(«.«+)... ,-(ç)f(t(«) ..... (ç«+) ..... ï r(«p+l).., r(«)ï(l(«),.. = (Zr(«p)...r(«l)f(l(«l) ..... l(«p))) "Op+l. Thus kP+LOdAf = dcq2P°f and the result follows. For g PN(C, xB), we define dpg P/'(A..\-) by ((I)g)(c/1 ..... c/p) = Z G10"" 0 1 1 o wl'it.ilig the comodule structure indices as supel'scripts, Lemma 5.6. Oi, g e PNP(A. X) and d_4(Oi, g) Pro@ For b B, we have sincc AA(bal)= OE haï ® a l. Thus condition (5.1)is satisfied. Also, ( Opg) (a ..... aih, ai+l,..., __ Z(lï.. 0 0 0 1 llp) -- a i bai+ l .. ap9(a,.... = (dPg) (al ..... [] 2766 DONALD V . BARNES Thus (5.2) is satisfied for ail i and g E I)NI)(A,X). Since g(a ..... @) X , (9) (al .... , avb) .... %b(1,...,) = aï.. 0 1 aç)b ...... ap)b ap9(al,..., 9(al and it follows that dag P+INp+I Lemnla 5.7. For 9 PN(Ç, xB), PO Pro@ For any .-linear flmction t C X, setting u(a a' c) = aa't(c) for a. a' A and c CdefilleS a .-linear function a A @B A @ C . Bv Lemma 2.(a), r(«)(c)°tq(«) ) = .,,( r(c)/(«)0 (c)) = = F(cl)I(cl)t(c2) = ¢(cl)t(c 2) by Lemma 2.1(c) = t(c). Using this with t(c) = l(c2)°.., l(%)°g(c,/(c2),..., l(cp)l), we have (çP0)(C1,... Cp) = '(Cp)...r(Cl)()(/(Cl) .... = r(Cp)...r(Cl)I(Cl)O...l(ep)°(l(Cl) 1 .... ,l(cp) 1) = r(%)...r(c2)l(c2) ° . ..l(%)°g(c,l(c2) ..... /(%)1). Repeating this argument gives the result. Lenlma 5.8. If f PNV(A,A ") and dal P+I'p+I(A,ç). then po f = f. Pro4 Setting u(a a') = ar(@_l)...(l)f((l) ..... (-1).') defines a .-linear function u" A @B A X by Lenlma 5.3(b) and condition 5.2. have 1Repeating this argument ai)r(ai))...O 1 ,-(l)f«(l),....,(1)) 0 0 1 l(alp)) ai)-lZ(apr(ai)) @B @-lU(1 ®B ai)) by Lenlma 2.1(b) E aï " " " ai)--lO ir(a ..... ) (I)(,(I),.. ,/(apl--1), ai))" gives t.he result. [] Corollary 5.9. The ri I)° are isomorphisms. Pro@ Eï ° is the set of f I)N(A,X) with dal I)+INI)+I (A, x), and i11)° is the restriction i)01Eï ° --' I)N(C, XB). By Lemma 5.7, it is surjective and, bv Lemma 5.8, it is injective. [] SPECTRAL SEQUENCE CONSTRUCTORS 2767 Theorenl 5.10. Suppose = A/fil is projective as .ff-module a'nd that A/B is projective as lift B-module. Tben (NA,F,I, ) is a spectral sequence coustru«tor Pro@ have to show that the couditions (1), (2), (3) of Definition 3.1 are sat- isfied. Since "NA (X) = Holn(@(.), X) = Hom(.. H,,ln(@ - .4, X)) and . is .-projective, N is an exact flmctor. A flmctiou f ¢ n satisfies the further couditiou f(ba .... ,a) = bf(a,...,a) for all b ¢ B. In particular, f(a,...,a.) = 0 ifa ¢ B. Thus nNÀ (X) = Holn(n A, HomB(A/, X)). Thus nN is an exact flmctor. Similarly, (X) = nom(.4, no,,,(A/, X)) and by iuduction over p, N is exact. Thus condition (1) holds. Let Q be an i0ective A«-mo(hfle. Then H([4Q) = 0 for n > 0 by the usual theory of the nornmlised standard complex. X have to show that u ¢',v O'-x[ + O) = 0 forq>0. But Hç(°NPAQ/°N+Q ) = EïA'A(Q) PN(C, Hç(B,Q)) by Corollary 5.9. But Hç(B, Q) = 0 for q > 0 since Q is injective as B««nodule. Thus conditiou (2) holds. Sin«e F(}')= N(C, }'), «oudition (3) holds. [] 111 the discussion of the filtered nornlalised complex in Barnes [1. Chapter IV], the corresponding extra assumption that A/B be projective as right B-module was needed. If in the HS context, .4 is a Hopf algebra, then it cau be regarded as a A//B-COnlodule algebra. The Guichardet filtration is not the saine as that given by Hochschild and Serre, but by the result of the next section, the two filtratiollS give the sanie spectral sequence. t3. UNIQUENESS OF THE SPECTRAL SEQUENCE As in BarlleS [1, Chapter X], we construct for each cardiual a, a coiffe fmlctor which, restricted to the subcategory .A of objects of cardinalitv less than a, is a spectral sequence constructor. (This use of the subcategory .A is necessary because a coiffe functor with injective lnodel /I aud injective basis (M, U) only has the desired properties with respect to lnodules embeddable in M.) From the existence of this cofree flmctor, we deduce as in [1, Chapter X], that all spectral sequence constructors construct the saine spectral sequence. We need one technical lemlna to get around the diflïculty caused by C-modules not being A%lnodules. For this. we again need the assumption that ri is q-projective, that is, that there exists a .ff-module homomorphism e : A -- q with e(1) = 1. Lemma 6.1. For every C-module Y, tbere exists an ijective A«-module Q such that Y can be embedded in Q. 2768 DONALD ,V. BAF{NES Pro@ We first lnake Z = Hom(A, Y) a C-module l»y defining (f. c)(a) = f(a)c for c C. a A and f Hom(A. Y). X construct an embedding i : }" + Z bv setting (ig)(a) = e(a)g for g Y and a A. S« defined, i is a C-module homomorphism, because (i(c))(«,) = ()9« = («(«,))c = (@(,,)« = ((+) «)() tbr g E G, a E A and f E Hom(A. }'). It is clearly injective. Next, we use the standard embedding of Z in the coimhlced C-module Il = Hom.(C, Z), defining « : Z + II l»v setting a(z)(c) = zc for z E Z and c now ha.ve an embedding of Y in Ho»re(C, Hom(A. }')). By Lemma 2.4. we bave an embedding of }" in X B where X is the coinduced A:-module Hom(A «, }'). Taking any embedding of X in an injcctive A'-module Q, we get an embedding of use the thcory of cofree filnctors develol»ed in [1. Chapter X]. Out spectral scquence constructors consist of two ftmctors and txvo natural transformations in- stea<l of the single flmctor used in the HS context. To accommodate this. we shall sav that the pair (F, F) of fimctors. F defined on A and F defined on C. is simple «ofrec on the 1)asis (5I. r, V) if F is cofree on (M.U) and V is cofree on ((M). V). Theorem 6.2. Let C be ( Hop.f «l.q«bra over .q and let A be a rigbt C-comodule algebra with B = 4 `c. Suppose ,4. B and C are .-projective atd that A/B is C- Galois. Let ,,4 be the categorg of left A"-modules, C tbe category of right C-modules atd let 73 be the categorg of .-moddes. Let : ,A ---+ C atd ¢ : C -+ 73 be the flmctors defined bg ¢(X) = X B and '(}') = I "c for X ,,4 atd Y C. Suppose .4/.ql is .q-projective ad tbat .4 is both lefl attd right B-fiat. Then. for atg cardinal n. there exists a simple cqfree pair (T. F) with ijective model M ad injective basis (M. U, I'). and atural transformatiots 1.7 such that on the subcategory +-l of objects of gt of cardimlit.q less tha . (T. F, q,"f) is a spectral sequetce cotstructor for (¢, ). P.v@ By replacing bv a suitable larger limit cardinal, we nlay suppose that everv object in A¢, bas an injective resohltion in A, and that every object of C likewise has an injective resohltion in C. There exists an injective module X in A such that every object of A can he embedded in X. Likewise, there exists an injective module I" in C StlC}l that e»vry object of C can be embedded in Y. Bv Lemma 6.1. there exists an injective lnodule Q in M such that }" can be embedded in (). Putting Al = X @ Q. we obtain an injective module M such that every module in A has an injective resolution, all of whose terres can be embedded in M, and every module in C+ h an injective resolution ail of whose terres can be elnbedded in (M). By [1. Lemma X.3.2. p. 101], there exists a silnple cofree functor F from C to cochain complexes in wih basis ((M), V) for some injective I', and natural transfornmtion : H'(F) + R' which, on Ca, is a natural isomorphimn. The construction of T and the proof of the result now follows exactlv as for [1. Theorem X.5.3, p. 107]. Theoreln 6.3. Let C be a Hopf algeb'ra over . a'td let A be a rtgkt C-comodule algebra with B = A c. Suppose A. B ad C are .-projective and tbat A/B is C-Galois. Let A be tbe category of left A«-modules. C the categorg of right C- modules a'd let 73 be tte category of .ff-modules. Let c : .,4 -- C otd l, : C -- 73 be tbe flmctors defited bg d)(X) = X atd /,(}-) = }-c .for X A and } C. SPECTRAL SEQUENCE CONSTRUCTORS 2769 Suppose A/.¢I is .-pvojective and that A is botb left and right B-fiat. Suppose I F = (F, F-, I', 7') and F' = (F', F,, I-, 7-) are spectral .seque'nce co'n.structors for (¢, ). Tttea F and F cottstruct caoztically isomorpbic spectral sequences fTom the E2-1evel o,ward. Pro@ The mgument of [1, Thcorem X.5.4, p. 109] al»plies unchmged. EFERENCES [1] D. \V. Barnes, b'pectral seqvence constructors in algebra and topology, lk|em. Amer. Math. Soc. 53 (1985). Mt 86e:55(132 [2] H. Cartan and S. Eilenberg, Homological Algebra, Princeton University Prcss, 1956. Mt 17:1040e [3] A. Guichardet, Suites spectrales à la Hochschild-Serre pour les produits croisés d'algèbres et de groupes, .I. Algebra 235 (200l), 744 765. M1R 2001m: 16013 [4] H.J. Schneider, Representation theory of Hopf Galozs extensions. Hopf algebras, Israel .. Math. 72 (199()), 196 231. Mit 92d:1647 [5] D. Stefan, Hochschild cohomologg on Hopf Galois extenston...1. Pure Appl. Algebra 103 (1995), 221-233. MR 96h:16013 1 LITTLE \VONGA ROAD. CREMORNE NS\V 2(19(. AI'STRAI.IA E-mail address: donb@netspace.net, au TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2771-2793 S 0002-9947(03)03265-3 Article electronically published on February 25, 2003 FORMALITY IN AN EQUIVARIANT SETTING STEVEN LIIJLYWHITE ABSTRACT. ,Ve define and discuss G-formality for certain spaces endowed with an action by a compact Lie group. This concept is essentially formality of the Borel construct ion of the space in a category of com n|utat ive different lai graded algebras over / ---- H*(/3G). These results may be applied in computing the equivariant cohomo]ogy of their ]oop spaces. 1. INTRODIICTION In this paper we consider G-Sl)aces and give formality results for them in an equivariant category. More si)ecifically, given a G-space 31. we discuss fornml- itv of the Borel construction EG x G 3[ or, eqlfivalently, formality of the com- plex A(11I) of equivariant differential ferres. However, in the equivariant settiug, the mai) 111 -- {pl.} is rei)laced 1)3" EG XG .I -- BG, and consequcntly ail the comnmtative differential gra(led algebras involved are naturally R-algehras, where R = H ° (BG). Thus formality may be considered in the category of comumtatiçe differential graded R-algehras. We shall also consi(ler the augmented case, corre- sponding te eqlfivariant base I)oints, which are the saine thing as fixed points of the group action. We should like te call a G-space 11I %quivariantly formal'" when its Borel construction is formal in the above sense. However, the terre "eqnivari- ant formalitv" has corne te be used te describe the degeneration of the spectral sequence of the fibration JII -- EG xE 11[ -- BG, owing te the i)ervasive influence of [11]. Thus xve shall adopt the terminology "G-formal" in this paper. We give seine general results concerning G-formaliy of products and wedges and reductions te subgroups. This is followed by several examples of G-formal spaces, including compact Ki4hler manifolds and formal elliptic spaces, among others. Of course, we rnnst make appropriale assumptions on the G-actions of these spaces for the results te hold. As an application of these results, we compute the equivariant cohomology of loop spaces. (If 11I is a G-space, then se is the loop space of 11I in thc obvious way.) Our motivation cornes frein considering the cohomology of symplectic quotients of loop spaces, sec [18], although the results are of general topological interest. shall use an "'equivariant" bar complex te compute the equivariant cohomology of the loop space. If the G-space 11I is G-formal, then the bar complex, which is generally a double complex, loses a differential and becomes a single complex. allowing for seine casier calculations. In the last section we compute an exmnple. Received by the editors January 1, 2002. 2000 Mathernatics Subject Classification. Primary 55P62; Secondary 55N91, 18G55, 57T30. Key words and phrases. Rational homotopy theory, equivariant cohomology, bar complexes, loop spaces, homotopical algebra. @2003 American MathematicaI Society 2772 STEVEN LILLY%'H ITE In an appendix, we discuss bar colnplexes and Eilenberg-Moore theory concern- ing the pull-back of a fibration. \Ve also consider equivariant versions of these results, which are used in several of the proofs in the lllain body of the paper. In what follows, we shall generally assulne that G is a compact. COlmected Lie group and that all spaces are colmected. Whenever we need to use the localization theoreln iii equivariant coholnology, we shall assulne that the spaces under con- sideration are of the holnotopy type of filfite-dimensiolml G-C\V-colnplexes, and flu'therlnore that they have finitely lllallv COlmective orbit types, lnealfing that the set {[G°] ].r e /I} is filfite, where Gx is the stabilizer subgroup at x. G°x is the comected COlnponent of the identity, and [G°] denotes the set of conjugacy classes iii G. This latter condition is autolnatically satisfied, bv the way, if ,I is colnpact or if G = S 1. I would like to extend lny appreciatiol to Chris Alldav. who took the tilne to read the lnalmSClipt and offel'ed advice on several key points. In particular. Proposition -1.7 is due to hiln. "2. /«C)Ç.A AND FORMALIT5 In this seçtion we recall SOlne ilnt»ortant facts al»out the category of COlmnutative differential graded algeblas, the notion of forlnalitv, and the colmectioll with ratio- nal holnotopy theory. \V shall assume ibr now that out algebras are /«-algebras, where t" is a field of charaeteristic zero. \Ve shall denote l»y/«CT)Ç ° the category of COlmnutative diflerential graded /¢-algebras that are concentrated in non-negative degrees and have a differential that raises the degree by one. \Ve assume further that H°(A) k, for ail A in hC7)91 '. \Ve shall denote bv /«C7)9 the category of algebra8 in /«CT)ÇI ° that are auglnelted over t" (i.e., there exists for each A a lnap " ,4 --/«, with/« COlCentrat.ed in degree zero), together «ith auglnentation- preserving lnaps for morphislns. \Ve shall ca]l an object of hCT)ÇI (resp. kCT)Ç.A °) a/,'CDGA (resp. kCDGA°). We recall Quillen's abstract approach to homotopy theory, [22], [23]. He begins by defining the notion of a closed model category. A closed lnodel category is a category, C, with 3 distinguished classes of lnorphislns, called cofibrations, fibra- tions, and weak equivalences, which satisfv a nulnber of axiolns. The holnotopy category, Ho C, is defined to be the localization of C with respect, to the class of weak equivalences. Quillen introduces a notion of homotopy and shows that Ho C is equivalent to the more concrete category ho C which has for its objects the cofibrant/fibrant objects of C, and for its morphisms the homotopy classes of maps. We point out the importmlt fact. that two objects X and }" in Ho C are isomorphic if and onlv if there exists a chaill (in C) of weak equivalences (1) In [4] it is shown that the categories kC7?(ï,,4 ° and kCDgI are closed model categories where the weak equivalences are the quasi-isolnorphisms (maps that ill- duce an isolnorphism on cohomology), fibrations are the surjective morphislns, and cofibrations are lnaps that satisfy the following lifting condition: a lnap f is a FORMALITY IN AN EQUIYARIANT SETTING 2773 cofibration if for everv commutative diagram with p a fil»ration and weak equivalence, there is a mal» ffoto }" to 1: making the diagram commute. (Actually, in [4], the authors do hot assmne that H°(A) for all algebras A. have includcd this assmnption for case of l>resentation, but the difference is slight.) Given a closed model category, C, with initial object ,, an object B is called cqfibrat if the map B is a cofibration. B is called a cofibr«mt model for 4 if B is cofibrmlt and there exists a weak equivalence B + A. It follows ffoto the axioms for a closed model category that everv ol»ject in a closed model category has a cofibrant model. Moreover, therc are various lifting and homotol>y results associated with cofibrant algebras: see [4], section 6. X> mention one here. If ç : Bx + B2 is a quasi-isomorl>hism, an<l we have a mat> f : A Be with A cofibram, then there exists a lift f : A + Bt such that çf oe f, where m denotes homotol»y. Note that L'CÇM is I>ointed with point o]»je<'t L'. The homotopy groups of a kCDGA A are defined to be def (A/(4) ) nA = H , where , = ker, for : A k a giveu auglnentation of A. If f : B B is a weak equiva]ence of cofibrant kCDGA's, then f. : 'B 'B is an isomorphism. Tlms, if we define R'(A) a (B) £r B a cofibrant model of A. then fl'(A) is well-defined up to isomorphism. Moreover. if f : Ai A2 is a map of kÇDGA's, then f induces a unique homotopy class of maps f : Bt for fixed choices of cofibrmt ttlodels Bi. B2 of .41, -42. respectively. It follows that there is a unique mal» f. : H(A1) H(A2). Thus H is flmctorial, and different choices of cofibrant models yield naturally isomorphic such functors. In kCÇM, there is a special class of cofibrant models called minimal models. A minimal model of an algebra A is defined to be a. cofibrant model, , A. that is connected (0 k), and such that the induced differential on /(«]2 is zero. It can be shown that each algebra in #CÇM has a minimal model, unique up to isomorphism. If BI is a path-connected topological si)ace, the (pseudo-dual) homotopy groups of AI are dened to be H'(M) d fl,(A.[AI) ) = (,), where is a minimal model for A'(M). Here, A'(M) denotes the Sullivan-de Rham complex, which is a QCDGA; see. for example, [3] for the definition. If M is a smooth manifo]d, we mav also use the ordinary de Rham complex, taking k to be Halperin has explicitly identified the cofibrations (and hence cofibrant objects) in #CÇM. Çofibrations me the so-called KS-extensions, and the cofibrant objects are the KS-complexes. Since these llotiotls will be important to us, we give their Definition 2.1. A map f : A B of kÇDGA's is said to be a IçS-e'lesion if there exists a well-ordered subset C B = {a'o}, Sllch that .4 C I (£) is an isomorphism of commutative graded algebras, where (£) dent»tes the 2774 STEVEN LILLYWHITE graded comnmtative algebra on the set E, and the map is induced bv f and the inclusion of E C/3. Identifying/3 with A (E). the differential on B satisfies (1) dB(a @ 1) = dA(a) @ 1, (2) dB(1 where E< = {x [[ < }. If E also satisfies deg(x) > 0 Vx Ç E, and deg(x) < deg(x) fl < a, then f is called a minimal A'S-extension. If A = k, then we replace the word "extension" by the word "complex" in the definition, obtaining the notion of KS-conple. (A minimal KS-complex is the smne thing as a minimal algebra, defined above.) A minimal KS-extension in which A is also minimal is called a A-'minimal A-e¢tension. Note that in a (minimal) KS-extension. A(E) is a (minimal) KS-complex, with differential such that ¢ 1 :A A(E) + A(E) is a map of kCDGA's, where ¢ is the augmentation of A. Moreover, all of these maps mav be ruade compatible with augmentations. If A is a kCDGA, then its cohomology, H(A). may be considered to be a kCDGA with zero differential. Definition 2.2. ,| is said to bc for'mal if A H(A) in Ho(kCl)Ç.A). It is easy to see that this (lcfinition is eqnivalent to the following two. Lemlna 2.3. Consider tbe cate.qory kCÇ. The following are equivalent: (1) A is formal. (2) Theve is a diagram A B H(A), where the maçs are weak equivalences and B s a coflbrant modelfor A. (In particular, we may pick B to be minimal.) (3) There is a chain of quasi-isomorphisms A *- A1 -- A2 *- "" - A, -- H(A). This theory has an important application to rational homotopy theorv. It turns out that the homotopy category of rational finite Q-type nilpotent spaces is equiv- alent to the homotopy category of the fllll subeategory of QC/PÇM consisting of algebras A with HA of finite type, [4]. Thus we may "do'" rational homotopy theory in a category of differential graded algebras. As an example, if X is a path-commcted, simply-connected topological space of finite Q-type, then there is a natural isomorphism II'(A°(X)) Ho,n(r.(X) ®Q, Q), where A°(X) is the QCDGA of Sullivan-de Rhmn differential fonns on X. If X is a smooth manifold, the saine statement for homotopy groups holds if we use instead the de Rham algebra A°(X) and replace Q coefiïcients bv NI, or C. There is hot a corresponding equivalence of homotopy categories over NI or C, however. A path-connected topological space is said to be formal if its Sullivan-de Rham algebra A°(X) is fonnal. If X is a smooth manifold, we may use the de Rham algebra and real or complex coefficients. However. a well-known result in rational homotop.v theory states that formality over NI or Ç implies fonnality over Q: see, for exmnple, [15]. FORMALITY IN AN EQUIVARIANT SETTING 2775 Formal spaces include comI)aCt Kfihler mmfifolds and many homogeneous spaces, including compact globally synlmetric sl)aces. Produçts, wedges, and Comlected smns of formal spaces are agaill forlnai. The top«,logical consequences of formality include the vauishing of ail Massey prodnçts. Moreover. the rational homotol)y type of suçh a space is deternfined solely 1¢« its cohomology algehra (at least for a large class of such spaces). 3. RC)ç AND G-FORMAIATY In this paper, we shall Ie concerned with equivariant versions of standard for- mality resuhs. Let G he a compact, COmlected Lie group. Then H'(BC:h) is isolnorphic te the hCDGA freelv generated bv a finite number of generators of even degree. shali denote H de H'(BG). dt, fine the category CÇM ° te be the category of comnmtative differential graded -algelras. shall contimw te assume that H°(A) k for ail algebras A. Tlms, we ohtain a faithflfi forgetfld fimctor ff'oto RCÇA te kCDÇA". also define RCÇA te l»e the category of commutative differential graded R-algelras augmeuted over R. Conlposiug aug- mentations with th¢' augmentatiol R h. we get a faithfifl forgetfid fimctor fiOnl RCDÇA te L.CDÇA. It is a standard result that if C is a closed model category and B is au object of C, then the "'over category" C/B whose o]»jects are nlaps X B and whose lnorl)hisms are commutative squares of the type X f " 1 1 may be given the structure of a closed model category with the following definitions. Such a morphism in C/B will be called a fil»ration, cofihration, or weak equivalence, if the map f : X I" is such in C. A similar statement holds for the %roder category", BC. See [7] for these and other results about closed model categories. Thus we see that both RCDç = RkCDçA and HCDç = RCDç/H are closed model categories. Moreover, the sinlplicial category structure on kCDÇ defined in [4], section 5. induces a simplicial category structure on RCDA" and HCDÇA in such a way that the results of [4], section 5, suitably modified, hold for these categories as well (cf. [22], II.2. proposition 6). Frein this, it follows that the homotopy results of [4], section 6, suitably modified, hold for RCDçA ' and RCDç as well. Definition 3.1. shall sv that an RCDGA (resp. RCDGA ) A is fowml if A H(A) in Ho(RCDç) (resp. Ho(RCDç)). If a filnctor j : C C t)etxvn two closed model categolies preserves weak equivalences, then A" l" in Ho C iluplies j(X) j(l') in Ho C. Thus if an algebl'a A is formal an RCDGA, then it is formal as an RCDGA , and a kCDGA, etc. Sul)pose a smooth lnanifold BI has a smooth action of a compact Lie group G. The equivariant coholnolo&v of I mav be computed bv means of the Cartan complex of equivariant differential ferres: A;(M) = ((S* A'(M)) G, dG) where the different.ial, da, is zero on S*, and for o. A" (M), dGt = de-uitx, o, where 2776 STEVEN LILLYVHITE the {Xi } are fundamental vector fields of the action correspmding to a basis of l- and the { u, } are the corresponding algebra generators of S*. which are given degree two. If 31 is just a topological space, we llltly compute the equivariant cohomology of 3I bv means of the QCDGA A(3I) of [2], when G = S t. Alternatively, ce could use thc de Rham algebra of the Borel construction. A'(EG xa M) when M is a lnanifold, o1" the Sullivan-de Rham algebra of the Borel construction when M is hot a manifold. shall let A;(M) possibly denote any of the above kCDGA's. leaving it to the reader to interpret which mdel one prefers to use, xvell as whid gromM fiekl k. FLra çOlnl)rehensive tleatment of equival'ialt de Rham thcory, see [:]. Using either model, it is olvious how to oltain an R-algelra structure on A(M). i If is indu«cd 1,v H ,t(pt.) A(M), »vhere the fil'st map is a choice of mildlnal model for A(pt.) in kCÇ, and the second map is induced from the map 1I {pl.}. If vc use the Cmtan models, then the algebras A(M) are manifestlv augmented (ver H whcn the group action bas a fixed point. This is lecause in the Cmtan model, A;(pt.) = , and thc inclusion of a fixed point gives a mal» A;(II) A;(pt.) = . [h,wever. if we use the Sullivan-de ham complex ff the Bnel «mstruction. then A(pt.) = A'(BG) ¢ R. Thus we nmst use a quasi- isomorphic «omplex that is Slnaller and augmented over . In [1], Allday shows that the «Oml,lex ,/-()is quasi-isomorphi« to A;(M), where ,/ A(3I) A(pt.)is induced by the in«lusion of a fixed point into [, and is embedded in A(pt.) via i as above. Clearly. tl-() is augmented over R. and is flmçtorial for eqtfivariant maps of G-spaçes. We shall almse notation and continue to write A;(M). even when we may really mean I-(R). Let ÇTO denote the category of l)ath-comected topological G-spaces with morphisms the equivariant maps. Then the under category {pt.}ÇçO consists of "%ased G-sl)aces", which is the saine thing as G-spaces with non-empty fixed- point set and a choice of base point in the fixed-point set. Then A(-) gives a flmctor from ÇçOç to RCDÇ and from {pt.}ÇçOç to RCDç. Definition 3.2. We shall sav that a G-space .I is G-formal if A(3I) is formal as an RCDGA". A G-space M with equivariant base point p (i.e., a choice of fixed point p 3I a) is G-formal af p if A(3I) is fornlal as an BCDGA. where 4(M) is augmented via the inclusion of p into If we continue to define a minilnM model of an BCDGA as a connected cofibrant model M for whMl the induced diflbrential on ker e/(ker ) is zero, where is an augmentation over . thell there moEv hot be a minimal model for everv algebra in C. An exmnple is .ç actiug bv rotations of .q2 about an axis. It is easy to see that there can l»e no minimal model for A. (S ) in BCDÇA. However. there is a fairly canonical choice of cofibrant model for an CDGA. Let A be an RCDGA . Then the map R A, viewed in kCÇM, mav be factored as ,M A with the first map the inclusion, the latter lnap a quasi- isomorphism, and M a minimal KS-complex, [14]. Note that the differential on R @ M is hot the tensor product differential: see the definition of a KS-complex (Definition 2.1). The map R R G' . is a nfinimal KS-extension. in particular a cofibration in kCçM, and hence we sec that R , . is a cofibrant model for A in RCÇM . Suppose A is, moreover, an algebra in RCDÇM. and let e : A R be its augmentation. Then composing B ï . A G B gives an -augmentation for FORMALIT' IN AN EQUI\%RIANT ,qETTING 2777 R .A4. Thus, R . .,M/ becolnes a cofibrallt model fbr 4 in the categorv RCTbÇ.A. As defined, itis unique up te isonorphism. Fol" those algebras of the tbrm A(./II) arising ri'oto a group action on the spa«e 31. this cofibrant model is more explicitly given b) the Grivel-Halperin-Thomas theoreln, which states that there is a c(mmmtative diagram (z) i R --* R '.in,..t --, asso«iated te the filtration 11I -- EG xc M -- Bd7. where .A4 is a minimal model br M. and the ],otom row is a A-minimal A-extension. see [121, [14]. Definition 3.3. && shall rebr te R . the G-mod«l of A, or just simply as the G-mod«l of M. when A : A(M). Sometimes we mav ehoose te dent,te it lx _( df B. . Note that R , may tdl te lin minimal as a k('D(',A. Folloving [1], [3], given a lmth-colmectcd G-space 1I with equivarimt I,ase point (i.e., a fixed point) p. the equivariant (pseudo-dual) k-lumotopy groups are defilmd te (3) Hâ,p(M) a '(R ._ Al)= H'(kere/(kel'e)2), where ge : R . + R is the R-algebra augmentati«m induced ])x the inchisiol of p into M. as above. The assigmnent (31. p) (R..ç.«) gives a flmctor ff'oto {pl. }çOç te Ho(RCDçA). and t he equivariant pseudo-dual k-homotoW group are fimctorial as well. Note that if M is G-forlnal. then the equivaliant pseudo-dual k-homotopy groups are determilmd by the equivariant coholnology ring of 1I. The following lelmna is uscflil for comparilg the equivariant pseudo-dual k- holnotopy groups te the oldinarv pseu&-dual k-homotopy groups of the Borel construction. Lemlna 3.4. Let A he an RCDGA «td let R. .A4 t,«- the G-model for .4. Then R ,. )., is minimal in kCDgA. Pro@ We have the augmentatiol g : R ,.-_ a'vl --, R. vhich is a lnap of RCDGA°'s. The differential. D. on /7 ._a .A4 satisfies D(r. ' 1) = 0. for r R, and generally has the brln D(1 " oz) = r C 1 4- ri ai 4- 1 ._ç da, where a, ci, .,. r. ri R with deg(a).deg(a).deg(r).deg(r) > 0, and where d is the differential in «VI. New. I) = eD(1 ® a) = "r + rie(ai) + e(da). Since da (+)2. and e is an algebra map, it bllows that. ris(ai) + e(da) (17+) 2. Hence, we nmst bave that r = 0, and it fbllows that 17 .-> .k4 is mininml. [] As ait exalnple, the pseudo-dual k-holnotopy groups of the Bolel constrnction of S t acting 011 '2 do hot distinguish the trivial action froln a standard non-trivial one, whereas the equi'a.riant pseudo-dual /-honlotopy groups do. 4. C-ENERALITIES CONCERNING (ïLFORMALITY In this section we give some basic results about G-formalitv. ilwluding reduction to subgroups and the G-brlnalitv of products and wedges. 2778 STEVEN LILLYçVHITE \\ I)egin 1)y noting that formality in the category HCT)ÇA ° is equivalent to for- malitv in kCDGA, hl genera.1, for two R-algebras A and B, A B in Ho(kCÇA) does hot imt)ly that A B in Ho(RCÇA°). Nevertheless, we have the following. Lemma 4.1..4s.s'ume that R -- 4 is an RCDGW' and that we give H(A) the j* R-algcbra structure R -- H(A). Then A is formal in I,'CT?gA if and onlg if A is formal in RCDÇ.A ('. Proof. If 4 is tbrmal in RCT_GA", then it xvill be so in 1,'CTÇA, as we have noted al»ove. Let us now assume that A is formal in/,'CY)ÇA. Let ,'( lin a minimal model for A and let R 9 .'v/be the G-modvl for A. Thon we have a «onmmtative diagram of/'( DGA s (4) .4 Sin«e R, A is «ofibrant in kCI?ÇA, tllcre exists a nlap. which is ne«essarily a quasi- isomorphism, H.-o A -- N" making thc uppcr right square homotopy commute. This gives us a quasi-isomorphisn R O Ad -- .&oe -- H(A). Thon the map (5) (*)- r)* R®,H(A) -« H(R®.Ad)---,H(A) is a quasi-isolnorphisln and a mai) of R-algebras. R«mark 4.2. \Ve note that this is hot true for lnaps, however. That is, if f A -- B is a nmp of R(DGA s, and fis formal as a mal) of/,'( DGA s. then f need hot be a formal map of R(DGA s. In the category RcT)gA, fi)rlnality is a. conccl)t distinct fronl formality iii the category kCg)ÇA, hl fa«t, it is easy to see that .,I is G-fornml a.t p if and only if the map i " BG ---, EG x M is a formal map. where i is the map indu«ed bv the in«lusion of p into 3I. Definition 4.3. Sui)pose that G acts on a space M. Then the Serre spectral sequence associated with the fibration 31 - EG xa M -- BG is the saine as the spectral sequence (from E2 onwards) obtained ri'oto the G-lnodel R ® ,/ via the filtration ffP = R >-p ® flA. If this spectral sequence degenerates at the E2 terre. then [11] refers to 5I as being equivariantly formal. For obvious reasons, we wish to avoid this terminology; however, to conform as well to current trends, we shall sa3 that 31 is er when this spectral sequence degenerates at the E2 terre. Proposition 4.4. Let G act on a space M. Suppose that K C G is a cloaed. connected subgroup. If 5I is G-formal at p (or G-formal) and er. then 31 is K- formal at p (resp. K-formal). FORMALIT IN AN EQUIVAIIIANT SETTING 2779 Ptvof. We fil'st consider the case where 'I is G-folnml at p. The inclusion h" C G induces a pull-lmck diagraln Eh xtç 111 -- EG xG 111 i B Iç -- BG We shall denote H'(BG) by Re;, a.nd sinfilarlv for Hl. If we are using the Cartan complex of equivariant differential forms, then there is no lnobleln with the proof. If ce are using Allday's construction, q-l(H), as notated al»ove, then we face file possibility that this construction may hot be flmctorial with respect to changing the group. This is because there may hot exist choices of minimal models so that RG -- Rh -- .t'(BK) conmmtes with Re, -- .I'(BG) - -I'(BIç). Then there «ould hot exist an iuduced map q- (R) -- q- l (/ï't,)- This problenl may be circumvented by the following procednre, as pointed out to us by C. Allday. ('onsider file diagranl (6). Let f" &,note tlle mal»ping cylindcr of the top row, and Y the malpillg cylinder of the bottom row. Then we have a in which the nmps jl,fl induce SUljections on differential forms and the maps j2,. induce quasi-isonlorI»hisms on differential forms. Il is easv to show that we nmv use 17- to form the complex q-l(/ï'G), as discussed in section 3. and that this COlnplex will be quasi-isomorl»hic to ,4(1I), and G-formal at p if 31 is G-formal at p. Moreover, we now may obtain a COlnlmltative diagram A'(Y) A'(BIç) in which the vertical arrows are quasi-isomorphisms, since the inap A°(Y) -- A°(Btç) is onto. This follows bv the result for RCDg,A. which is the analog of By Lemma A.1 of the appendix, there is a qui-isomorphism of kCDGA's (9) (A'(Blç),A'(BG),Ab(M)) A-(M), where we are abusing notation in the event that we are using Allday's construction. Then, in either case, we obtain a quasi-isomorphisln of R/-CDGA's (0) [(RI«,Ra, A'a(M)) A-(M). The bar complex (10) is an R/«-algebra via the R/« factor, and has an RK-augmen- ration given by e(rK,ct) = rKï(eG(Ct)), where r/ ¢ Rê,', et ¢ A.(/I), and ea " 2780 STEçEN LILLh V HITE .4;(I) -- Rc is the augmentation of I for the action of G. Bv the asslnp- ti«m of G'-formality, we get a conmmting diagram whose vertical arrows are quasi- isomorphisms: (11) Rb RG -- Riç RG --" H(ll) Then xxe obtail, lhe folh)wing sequence «)f lllaps, which are st'en to be R,CDGA «luasi-isolmn'l»hislnS i W standard cmnlmrison theorems tbr their associated Eilen- ierg-g Ioore spec tral seq,ences: (12) B(R,ç. Ra, .4;(.I)) B(R,,. Ru..a(3I)) (R/,. Ha. H;(M)). Now the lmr «omplex B(Rt,. RG. ff;(.ll)) has mly lle singlc difl>rential . alld comt,utes TorRo (Rb. H(3I)). Since 1I is cf. Hç.(l) is a free RG-module. Hence we have thal (Rh,HG,II(3l)). is acvclic in lmr degrees greater than zero. and t ho projection to c()h()lllology (13) (Hr,, Re, H5(3I)). + ( Rç, Re, 1]:;(31))o H,-(M) is an RçCDGA quasi-iSOlnorphism. The case where we consider 31 to be G-formal in the category HGCÇ ° is similar. Corollary 4.5. Let G act on a space 31. Suppose that M is G-formal at p (or G-.formal) rmd er. Then 1I is formal in Pro@ Just take K to be the i(lentitv sui)gr()ul) iii Proposition 4.4. emark 4.6. If we use Remark A.5 of thc apI)elMix, then we can see that a lx-nlodel fbr M is given by Ra(Rh. RG,G(l)) : Rb R (I). In line with the general thelne of considering maximal tori in compact. COlmected Lie groups, we have the following face. which is due to Ç. Alldav. space M is of fiite tpe if H'(M) is a finite-dilnensional k-vector space for ail i. Proposition 4.7. Let G act on :I. a«td let T C G be a ma.rimal torus. If M is G-formal (G-formal at p). then M is T-formal (resp. T-formal af p). Moreover. if M is a space offlnite tpe. p M . and I is T-formal at p. then M is G-formal at p. Pro@ X can aheadv see tlmt G-forlnal ilnplies T-formal by the proof of Propo- sition dA. XX> only need the fact that now HT is a fi'ee Ho-module. which follows ri'oto the well-known fact that as Ho-modules. RT Ho Ç H'(G/T). Showing t.hat T-formel al p implies G-formal at p mv be achieved by imitating the proof that A K being forlnal ilnplies A is forlnal, for K an extension field of k. whi«h is corollal'V 6.9 of [15]. Olnit the details, bnt mention the setup. First. FOIRMALIT IN AN EQUIV4RIANT SETTING 2781 we see by I{emark 4.6 that a T-model for A(AI) is given bv/r T (-RG .A4c(M) with differential 1 C, De, where De is the difl'ereltial fiJr the G-mode| .Me(iii). Thus it suffices to show that if/-ff ®ha ,(/11) is formal as au RTCDGA. then .M(M) is forlllal as ail RG('DGA. It turns out t|lat the coustl'uctions of bigra(led and filtere(l models of the rçlevaut algebras, as in [1.5], givc lUodels iu the category HCTÇ4. Thç proof of corollary 6.9 may bc imitated xvitllout too llltlch difficulty. [] Proposition 4.8. Supl,oSe that X and arc G-spa«es. both of whi«h are G-formal (or as.smne X is G-fo'rnml at p and " is G-formol at q). and suplo.e that one or both of them is diagonal a«tiou of G. Proof. The pull-back diagram (14) G-fin'mal (resp. G-formal at (p. q)) for the x --, b,t.} gives rise to a lmll-lmck diagram EGxG(XxY) EGXGI EG xG X BG (5) O Then we o])tail a.li /( DGA qnasi-isomorphism by Lemlna A.3 of the apt)endix. If X and ] both have fixed points, then so will their product X x }'. In that case, 0 is a quasi-isolnorphisln of RCDGA's bv Lemlna A.3 of thc appendix. FurthCllnOlC, (17) - " " - " " B(A.(X)...a.()) Ai, " B(AG(X ), ,({pt.}), ) is an RCDGA ° (RCDGA) quasi-isomori)]fiSlU. Since X aud are G-forum.l. we get RCDGA ° (RCDGA) quasi-isomorphisms of bar cmplexes (H&(X),R.H&()) by standard arguments comparing the associated Eilenberg-Moore spectral sequen- CeS. Siuce one or both of X. ) is cf. just as in the proof of Proposition 4.4. the bar respect to the bar grading, and the ln'ojection to its cohomology is an RCDGA (RCDGA) quasi-isomorphism. 2782 STEVEN LILLYWHITE Proposition 4.9. Let X md I" b« G-spaces u,hose fi:red-point sers are non-empty. Picking base points p E X G and q l'G. we may form the wedge X V I" along these base points. Theu G acts on A V }'. If X is G-formal ai p and 1" is G-formal at q. then X V }" is G-formal al the join of p and q. Pro@ Let ex,e denote the augmentations of equivariant differential forms, and let ix, if denotc the inclusions of X, 1" into X V Y. Then Mayer-Vietoris gives a short exact sequence Thus ix + i- induces an isomorplfisn R Moreover. since ex, say, induces a surjection in cohomology, the associated long exact sequence splits into short exact sequences, and thus (2O) nS(X V ) n5EE(X ) e bO')" Since ç and k are G-formal. we have maps A(X) a(X) H(X) which are quasi-isomorphisms of RCDGA's, and similarly for I'. So we have a commutative diagram whose rows are short exact sequences: (21) o Ab(x) ¢ A.(',) --- Ab(X) ,4c(Y) --, o Then we obtain maps between the associated long exact sequences in cohomology. By the 5-1emma, it follows that the maps (22) A,(X) eR A,(Y) are quasi-isomorphisms. It, is easy to check that these maps are compatible with augmentations and the R-algebra structure, so are RCDGA quasi-isomorphisms. 5. EXAMPLES OF G-FORMAL SPACES In this section we give some examples of G-formal spaces. 5.1. Compact Kihler lnanifolds. Let 3I be a compact Khler mmfifold, and G a compact, connected Lie group acting on 3I by holomorphic transformations. introduce equivariant holomorphic cohomology groups. Since M is a complex manifold, the complex-valued differential forms on M are bigraded in the usual way. ë shall denote Sg* @ C by simply Sg*. Then we define the eqtfivariant Dolbeault. cohomology to be the cohomology of the complex (23) ([$9" Ce AP"(I)] ° ; + ,z,). Here Zi is the holomorphic vector field on M which cornes about by splitting the fundamental vector field Xi = Zi + Zi into its holomorphic and anti-holomorphic FO1R.MALITh IN AN EQUIVAIIANT SETTING 2783 colnponents. The gellerators ui E Sg* are given bidegree (1.1). The operatol's act in a Silllilar way as for file ordinarv equivariallt cohonlology. \\k' shall denote the qth cohollology of this complex by H'q(AI). The following theoreln was proved in [17] and ildependently established in Theorem .1. Suppo.s'e that Al is a compact KShler maifold edowed with a holowophic action of a compact, covected Lie 9roup G. and sui»pose that I s ef for the action of G. Then M is G-Jbrmal. IJ" M a fized point. Pro@ The Cal'tan COlnplex is (A(M),da)= ([S* N.4"(3I:C)] Let Xi = Zi + Zi be the splitting of the flmdalneltal vector field Xi into its hololnoll»hic and anti-hololnorphic parts. The differential d = O + 0 also splits. Hence we lnay split the equivariant differential as d + utx, = (0 + tz,) + (0 + ut,). The COlnplex [Sg*@ A(M;C)] a is bigradcd by giving bidegree (1.1). and taking the usual l»igl'ading on A* (1I; Ç). It is easy to show that (24) ([Sg*A'"(M:C)]a ; (+u,z,),(O+u)) is a first qua.drant double çOlnplex. Accoldingly we have two canolfical filtl'ations of this COlnl)lex. claire that the spectral sequences COl-lespondilg to both of theln degenel'ate at the E1 terln, and lllOl'çover are n-opposite, memfing that 'F p H' for p + q- 1 = tt. Formalitv for 4(M) then follows owing to the results in [6], sections 5 and 6. Let us consider the filtration iii which we take + uitz, coholnology first. This is the Dolbeault equivarimt coholnology defined above. It itself fOl'III 3 fil'St quadrant double complex with the two differentials and uitz,. Let us filter so that we take the 0 coholnology filSt. Then the E terin for the equivariant Dolbeault COlnplex is (additively) (25) H([S* 4"(M)]a:O) [H(Sg* A'(M):O)] a (Sg H(M)) a (26) (SO*) a H3(M) H'(BG) HO(M ). Now bv ordinary Hodge thcory fbr COlnpact Khler manifolds, this lt is isolnorphic to H'(BG) H'(I). But now there can be no further non-trivial differentials iii the spectral sequence, by the aSSUlnption that M is ef. This result follows mmlogously for the other filtration, which is just the COlnplex conjugate of this one. Furtherlnore, it is easy to see that the two filtrations m'e n-opposite. Hence we have a "OGOG-lelnlna'" for the equivariant differential forlns, where we mean by 0G the equivariant Dolbeault opcrator as defined above. Forlnality follows via the sequence of CCDGA quasi-isomorphisms (27) Ab(M) ker(Oa) Ha(M ), which are the inclusion and projection, respectively. These maps are lnaps of R- algebras, and moreover, it follows that for equivariant holomorphic maps between M and N, we get a commutative diagraln linking the sequence (27) for .I to the analogous sequence for N. In particular, if the action of G on M bas fixed points. then the inclusion of one (chosen as an equivariant base point) gives augmentations so that the sequence (27) comlnutes with auglnentations. That is, 3I is G-formal in RCÇA. 2784 STEVEN LILLY\VHITE Corollary 5.2. Sttppose that M is a compact K5hler manifold edowed witb a holomorphic action qf a compact, cone«ted Lie 9roup G. Assume that AI G # 7Jen M is Gçformal at ang fi.red point. Pro@ Lel p G 3I . Let T c G 1)c a maximal torus. Then 3I ¢ , and a theorem of Blanchal'(t savs that ;I is ef for the action of T: sec [9], Cai)ter XII. theoreln 6.2. By Theorem 5.1. M is T-fi»rlnal at p. By Pl-opositiol 4.7. 31 is G-formal at p. Rcmark 5.3. The proof of Th«or,ln 5.1 implies an cquival'iant Hodge decomposition 5.2. EIHptc spaces. re«all tiret an elliptic space 1I stmce su«h tlmt both tl'(M: k) and I" are finte-(limensonM k-vector spaces, whcre (M) = (I') is mhfima] mdel fi»r 1I. b shall nse the followh remflt of [19]. PropoMt}on 5.4 (Lui»ton). Let F E e fibr«dion i which F s formd «nd «lliplic. and is .fovmol che simpl.q-conne«ted. If th« Serre spech'al of the fihralion degenerot«s «t the E2 l«rm. the E is formd eIso. Theorem 5.5. Let 3I e az dlipti« G-poce If M is formel and er. the M is (;çformol. If ll G . lhe zI is G-formol ci ez.q .red poinl. Pro@ We lmx-e the filtration ,I SG XG :11 BG. Thon Proposition SA hnpfies t, lmt A.(,ll) i fin'mal as a k('DGA. çhe proof of Lupton's proposition works (a(iapthg to oto" situation) Lv finding a model for A(AI) of the form R.,ç that is bigraded a a kCDGA. Hcre, ç is the bgraded (nfinhnal) model of 3I. Elements of R arc in degrec zero for the second grading, so tha (R -)0 = - (,)0- It is shovn that wth respect to tlw second grading we have H+(H ,) = 0. and hence the projection to cohomology (29) R., M (R.)o H(3I) is a quasi-isomorphism. Çlem'lv. this is a map of R-algebras. Ioreover. if 3I then this mat» commutes with the augmentations over R. This follows because first the mal» R ;-) (R :-))0 commutes with augmentations. Second. since the augmentation e : R - R is a map of RÇDGA's, e(do) = ) for ail . so that the map (R , ,)0 H(AI) commutes with augmentations. Corollary 5.6. Let 31 be at elliptie space. Suppose that a tortt T acts on 3I u,ith 3I T . Sui»pose fltrther that one of the «ompo#e#ls of the ç.red-pomt set, Pro@ Since AI is elliptic, it. follows (via localization and loca]ization for equivari- ant rational homotopy [3]) that each component of the fixed-point set is elliptic and k(AI) = (AI), where k is the homotopy Euler characteristic. But Halperin h shown that for elliptic spaces the conditions H °dd = 0 and k = 0 are equiva- lent. and moreover such spaces are forma/. Thus 0 = (3I) = (I). Hence 31 is formal and H°dd(Al) = O. But this latter condition implies that 31 is ef. So we mav apply Theorem 5.5. Remark 5.7. Suppose G aets on a simply-connected space AI with non-empty fixed- point set. Then bv picking a base point in the fixed-point set, we obtain an action FOItMALITY IN AN EQUIVARIANT SETTING 2785 of G Oll the Sl)ace of |,ased loops in 1I, deuoted tM. Since the cohomology of Lll is fret, we see that t]l will be G-formal if t/l is er. (Lupton's proof could be extended to this case, as well.) If G = T is a torus, and M is elliptic, then the condition that t]ll is er is equivalent to the G-model R ).A4(M) being mininlal in the category RCDçA: see [3]. 3.3.15. 5.3. Miscellanea. Next we shall give a few extra examples of G-fornlality. Theoreln 5.8, Let 11 be a space u,itb minimal modcl « = /(I'). Sui»pose that d.r = 0 .for all x G I "'v«' such th(d deg(.r) < diln lI. Sui)pose fltrther that the civle S = T acts on M. that M i.s @ and lhal cach comportent o.f the fixed-poinl set is formal ad satisfics H°ad(jl T) = O. Then ll is T-formal al an9 fixed point. Pvof. Since M is ef. the Serre spectral sequence for the fibration AI -- ETXT3I -- BT degenerates at the E2 terre. (Note that by the localization the«weln, this implies that M w ¢ .) Bv a standard change of basis argmuent, we lllay assllllle that in the T-mo(lel (R.,'t//, D) wc havc D.r = (). for x I 'w such that deg(.r) < (lira 31. Let i : ll r '-- M &,note the inclusiou of the fi'<cd-l>fint set. Th('n wc have maps of/ï'('DGA's (actually, the algebras on the righ-hand side of the diagram d<) no satisy H ° = k, lmt t]tis will m)t l)resent any l)robh'lnS) (30) i A.(I) AI(M r) R ® cM(M) R® 3A(3I r) /4(M) HT-(M r) t e t/'(M r) whel'e/t is a quasi-isonlorphisnl sinçe 3I T is folmal. Since 3I is ef. the nlap i* is an injection. We claire that hi(R ®.A4(M)) ç i*(H-(M)). Since the maps are algebra maps. it suffices to check this on alget)l'a genelatol's. Since M is er. the localization theorenl shows that i* is an isolnolphisln in degrees >_ dira M. Also if c R-,M(3I) bas odd degree, then bi(o) = [k since H-aa(M r) = 0 bv assmnption. So it sutfices to check the claire on algebra generators of R N,/(,àl) of even degree less than dira M. Let o be such a generator. If o G R, theu the claire is obviouslv true. If o' G .A4(]l), then by assunlption Dc = 0. Then bi(o) = [i(cQ] = j([o]). Thus we have a map (31) which is a quasi-isomorphisn of RCDGA's. Corollary 5.9. Let ]I be a smtplg-connected space witk minimol model ](I/). Suppose that dx = 0 for all x Ç V « such that deg(x) < dira M. Suppose furtker that a torus T acts on M. that M is @ ad that each composent of the fi.red-poivt set is formal 2786 STEVEN LILLYVCHITE Pro@ First of all, there is a subcircle S t C T such that 11I si of this circle ,S 'l T induçes a imll-ba«k diagram: = AI r. The inclusion ES X s B,ç I -- BT Since the action of T is er, the Serre spectral sequence for the fibration on the right degenerates at the E2 terre. Bul then the saine is true for the pull-back fibration. Hence lhe S action is er well. Now the result follows from Theorem 5.8 and Corollary 4.5. Corollary 5.10. Let 3I 4 be a .sp,,ce such that Haa(kI) = 0 atd dimkl = 4. Eq»pose that a circle S = T acts on M. Ten M is T-Jbrmal al ang fi«ed point. Pro@ have that H°aa(I) = 0, so that 31 is er. Then ai T . Bv localization. H°aa(5l r) = 0. But path-comccted spaces xvith H = 0 of dimension less than or (,qual to 4 are tbrmal; so each component of 3I y is formal. The result follows bv Thcorcm 5.8. Remark 5.11. A simple example of an Sl-space satisfying the conditions of Theorcm 5.8, but which is hot Kfihler or elliptic, is the following. Let S act on S 4 so that the fixed-point set consists of two isolated points. Exlend this to a diagonal action of S t on S 4 x S 4. Then, removing a ncighborhood of a fixed point, we may forln the coimected sure S 4 x $4S 4 X S 4. This manifold then inherits an S action with 6 isolated fixed points. It is hot elliptic, and not even synlplectic, since H = 0. It is easy to check that it satisfies the conditions of Theorem 5.8. so is sl-formal. (This can also be seen by proving that the commcted sure (ruade in an equivariant setting) of G-formal spaces is again G-formal. which we have omitted.) conclude this section with two examples that do hot involve the condition of 3I being er. Lemma 5.12. Let M be a simply-co,nected compact manifold. Suppose that G acts freely on M ad dira G k dira I - 6. Then M zs G-formal. Pro@ Since G acts freely, M/G is a simply-comected manifold of dimension 6 or less. Hence M/G is formal [21]. So EG Remark 5.13. Suppose, iii the situation of Lemma 5.12, we have that dira M - 6 rank(G). Let T C G be a llaXillml torus. Then by Proposition 4.7. MIT is a simply-commcted manifold of dimension greater than 6 which is formal. Lemma 5.14. Let M be a simply-connected elliptic space. Suppose that G acts al- most Jkeely on M (meaning ail isotropy groups are finite), and rank(G) = -X(M). Then M is G-fomal. Pro@ Since M and BG have finite-dimensional pseudo-dual rational homotopy, so does EG x a M, as mav be seen by considering the fibration M EG x a 3I BG. Since G acts almost freely, H" (EG x aM) is finite-dilnensiolml as well. Furtherlnore, (33) (EG xa M) = X(M) + x(BG) = -rank(G) + ra.nk(a) = 0. Thus EG x a M is elliptic with = 0, so is formal. FORMALITY IN AN EQUIVARIANT SETTING 2787 6. AN APPLICATION In this section we give an application of G-formality. We will shov« that the conqmtation of the equivariant cohomology of loop spaces simplifies considerably when the spa«e is G-fin'mal. Let us consider a silnply-commcted sImce 31. Suppose that G acts on 31 with non-empty fixed-point set. Let p iii c; tre a choice of base point. Then we get an action of G on the loops iii I based at p, (M:p), which we shall oftell abbreviate as LI. Let P(ll:p) he the space of paîhs in M, based at p. Then we have the fitwation (34) Lll -- P(M:p) where rr is the lna I) sending a path "y(t) to its value at tilne 1, '3,(1). rhleover, the G-action induces a imll-back diagraln of fibrations (35) EG xc Lll --, EG xc P(Al:p) BG -- EG xo 31 Hence there is a quasi-isomorphisln of/?('DGA's bv Lemlna A.a of îhe appendix. Now the inclusion of {p} into P(M:p) followed by w is îhe inclusioll of {p} into M. These nlaps are equivariant, so induce their analogs Oll the Borel constructions. Hence we get an ÇDGA quasi-isomorphism (37) (. «b(*),) Proposition 6.1. Let g act o a simply-coectcd s'pace .I with non-empty fixcd- point set. so thot G acts on M. Suppose tbat I is G-formal. Then there is an isouorph ism of R-algebras (38) H(.I) TorZ,(M)(. ). Pro@ bave that .45(Lll ) is quasi-isomorphic to (B,.45(M),B ) (via a se- quence of BCDGA quasi-isomorphisms). The ssumption of G-forlnality mcans we have a comnmting diagram of R-algebras (39) R , « , « HS(h obtain RCDGA quasi-isolnorphislnS (40) Aa(M ) « b(R, Ab(M),R ) ,- (R, A4¢E(IU),R)-- 27t8 STEVEN LILLYV'HITE This follows bv stalidald comparisol theorelns for the Eilenberg-Moore spectral sequences a.ssociated fo the bal" complexes. Thus we have that Y(R. H,,Ill). Ri is quasi-isolnorphic to A_;(fLI) (via a sequence of/?ÇDGA quasi-isolnorphisms). But the cohomology of B(R. H.(M), R) is TorH;(al)(R, R). [] Re'm«u'k 6.2. We can alwa..vs choose any lesolutiol fo compute Tor. But we note that we may always use the bar resolution, and using Lemma A.4 of the appendix. we see that when ;'I is G-f(»rlnal. H;(tII) mav be COlnputed via the (single) COlnplex (41) (/gR(H. H5(3I), R; 6). flemark 6.3. We conld also ol)tain analogous lesults fol' the equivariant cohomology of the free loop sl»a('e L,1l. ï. AN EXAMPLF |n this section we COlnpute ail exalnl)le of le equivariant cohomology of tle »ase(1-h)ol) sl)ace using fle l(n'lnalize(1 bar c«nnl)lex over of elnark 6.2. 7.1. Example: S 1 aeting on ILq . The circle .b ' acts on the 2-sphere .b " by rotations about an axis. sav the z-axis when ,5 '2 is the unit sphere in N a. This action is hololnOrl»hic and Halniltonian. Thus by Theorem 5.1..q.2 is G-forlnal (G = ,5q). It is easy to show that the equivariant coholnology ring is (2) , (S: ») ['. ,]/( + 0(.- ,,), where the degree of .r and u is t, and R = k[u] acts as multiplication bv u. The fixed-point set, F, consists of the norfl and south poles, kk shall write F = {N,S}. Let lS u be loops based at the north pole. Then S acts on !S . Then the equivariant cohomology of the based loops, Hs, (lH). lnav be computed as the cohomology of the bar coml)lex (43) ([(»[-], (.«_ ,,). []) ; Let w 1)e the SVlnl)lectic form on S 2. Then .r is represented - the form w,- uf G A.,(S2), and u is rel)l'esent('d by the forln u G A.,(S2), using the (artal COlnI)lex of equivariant differential forlns. Hem. f is the molnent lnap which sends a l)oint on S 2 C N3 to its z-colnponent. Then the inclusion of the north pole {Y} into S 2 induces the auglnentatiol, H.(S 2) H}({N}) k[u,] sending x -u and u u. k omit the details of COlnputing the bar complex, but one finds without difficulty the cohomology generators (1. x. .... x. 1) in degree n for n odd. and (u /2. 1) in degree n for t even. Owing to the shuffie product structure on the bar complex, onc sees that. as an R-algebra. whel'e a'i is an indeterlnilmnt of degree i. Remar 7.1. In this examI)le, the lOllnalized bar COlnplex (R. H., (S). R) is actnallv isomOlphic to the k('DCA minimal model for £S x s, [2j2. which is (45) .s, ., s = A(..,...) (& = 0: &. @ = FORMALITY IN AN EQUIARIANT SETTING 2789 where the degrees of u and p are 2. and the degree of .r is 1. The isomorphism is given by (1,x, 1) .r, (u, 1) u. and (l.r,.r, 1) y. e'ntar" 7.2. In this example, the space ES 1 xs t]S 2 is hot fornml, implying that tS 2 is hot G-formal. hMeed. Massey products alomM. APPENDIX A. [AR COMPLEXES AND EILENBERG-IIOORE THEORY In this aplwndix we shall discuss the theorv of Eih,nberg and Moore concerning tmll-backs of fibrations. will also consider equivariant versions of these results. For references, sep [20], [241, or [8]. Let us sut)pose that we have a filnation F + E B and a map f X B, so that we obtain a pull-back diagram: (46) Then t he maps f* and p* lnake A ° ( X ) and .4 ° (E') (difl'erent ial graded ) modules over .4°(/3). Let us aSSulne that B is simply-come«ed. Then a theorem of Eilenberg and Moore asserts that there is an isomorphism (47) 0" TOl'Ao(B)(A°(X), A°(E)) H°(Ef). We mav use the bar resolution to obtain a resolution of. say, A°(X) bv A°(B) - modules. Since we are considering A°(-) to be the de Rham or Sullivan-de Rham complex, we will use Chen's normalized bar resolution, see [5] or [10]. More specifically, the bar complex is (48) X B(A'(X),A'(B),A'(E)) = 0 A'(X) ® (sA'(B)) ® A°(E), where the tensor products are over the ground field k, and s denotes the suspension functor on graded vector spaces that lowers the degree by one. Hence the degree of an e|ement (a,w ..... v,fl) is deg(a) + }-'î (deg(vi) - 1) + deg(fl), where c A°(X), a:i A°(B), and fl A°(E). Actua||y, the bru comp|ex is bigraded. We introduce the bar degree, denoted B(A°(X), A°(B), A°(E))o. The bar degree of an element (a,a: ..... a,',/3) is defined to be -k. The other grading is the normal tenso," product grading, the degree of ail element (ca. ' ..... '., d) heing k deg(a) + i= deg(cvi) + deg(fl). 2790 STEVEN LILLYWHITE There are two differentials (49) d(a. c0,..., c0k, 3) = + (.50) -ri(a. ..... ».) = of total degree +1: (c/a. 1 .... , k,/) /,. + OE(- 1)ei-'+l (0¢. c01 ..... cOi-l.d:i.oei+l ..... (-)(a.«. .,. k-1 + (-)«-'+(. ..... where ei = degc + (legw +---+ degwi - i. The differential 5 has degree +1 with respect to the lal grading, while the differential d bas degree +1 with respect to the tensor produçt grading. ()ne mav verifv that da + ad = 0. and we put D de___f d -F ( to be the total differential. With the given bigradilg, we get a double complex with the two difl'erelitials d aml ri, which giw,s rise to the Eilenberg-Ikloore spectral sequence. ('hen's norlnalized version of this bar eoln|lex is the following. If f ff 4°(B), let ,5'i(f) l,e the opelator on B(A'(X). 4"(B), .t'(E)) defined bv (51) Si(f)(a. COl ..... la)k,/ ) ---- (tq, kU 1 ..... kUi__ 1. f., ..... for 1 <_ i _< /« + 1. Let II be the subspace of B(A'(X), A'(B), A'(E)) generated bv the images of Si(f) and DSi(f) - Si(f)D. Then define (52) (A'(X),A'(B),A'(E)) dej B(A'(.\').A'(B),A'(E))/II. Then I| is closed under D, and when H°(B) = I« (B is connected), then I1- is acyclic, so that /)(A'(X), 4"(B), A'(E)) is quasi-isomorIhic to B(A'(X), A'(B). A'(E)). Notice that in the normalized bar con@ex there are no elements of neg- ative degree, and with out assumption that /3 is simply-commcted, we are assured convergence of the associated Eilenberg-hloore spectral sequence. The map 0 men- tioned above is induced by the ma l) (53) 0- B(A'(X), A'(B),A'(E))--, A'(E/), which sends all tensor products t o zero except fol" A ° (X)®k A ° (E), where the map is (a. [3) /5"0 A f*3. Note that 0(lI') = 0. so that we get an induced lnap (54) 0"/(A°(X), A°(B). 4°(E))-- A°(E.). The nornlalized bar complex may also be auglnented. The augmentation, e, nlai)s all elements of positive total degree to zero. The elements of degree zero have the form (f. g), where f ¢ 4°(X) and g ¢ A°(E). Then we define e(f,g) = ex(f)eoe(g) = f(xo)g(eo), where x0 and e0 are chosen base points in X and E, respectively, and ex,ce are the augnmntations of 4°(X), A°(E), respectively. If we choose base points so that the pull-back diagram above preserves all base points. then 0 is an augmentation-preserving map. The bar conlplex has a natural coalgebra structure. Since we are inputting kCDGA's to the bar complex, we also obtain a structure of kCDGA on the bar complex via the shuffte product. FORMALITY IN AN EQUIVARIANT SETTING 2791 More specifically, if (et 1 ..... (gp) and (bi,..., bq) are two ordered sets, then a shuf- fie r of (al ..... Op) with (bi ..... bq) is a pernmtation of the ordered set (al ..... bi,..., bq) that preserves the order of the ai's as well as the order of the ba-'s. That is, we delnand that if i < j, then a(ai) < a(aj) a,,a «(bi) < «(b¢). We obtain a product Oll B{A'(X), A°(B),A°(E)) by first taking the normal tensor product on the A ° (X)® A ° (E) factors, then taking the tensor product of this product with t.he shufl3e product on the A°(B) ®i factors. As usual, we introduce a sign (--1) deg(a)deg(/) whenever o is moved past/3. One che«ks that this product induces a product ol, Chen's norumlized con,plex,/)(A" (X),. 1 ° (B), A" (E)). as well. Thus we arrive a, t|le following lmnma, whosc proof is left to the reader. For more details, see [16]. Lemma A.1. A.'surne that we bave the pull-back diagvam (46), wh«re p is a fibva- tzon and 13 is sirnply-connected. Then the nornalized bar «omple.c is a kCDGA. Moveover, 0 : O(A'(X). A'(B), A'(E)) -- is a quasi-isomorphism of kCDGA "s. Remark A.2. We note that Chcn's normalization is flmctorial. That is. if we haxe a conmmtative diagram of kCDGA's then we get a map of kCDGA's/)(A, B1,C1) --,/)(Ae. B2,C2). We may fornmlate an equivariant analog of the bar complex. Let us consider again the pull-back diagram (46). If we suppose flrther that X.B. and E are G- spaces, and that f and p are equivariant maps. then we obtain a pull-back diagram (56) EGxX ï , EGxB Note that we are assuming B to be simply-connected, which in turn implies that EG xo B is simply-connected as well. We ma- apply Lemma A.1 to the diagrmn (56). However, the bar complex (A(X), A(B), Ab(E)) has the extra structure of an RCDGA ° or RCDGA, depending on fixed points. We mav give it an R- algebra strucure via the R-algebra structure on the Ab(X ) factor, and we define the attgmentation as above, assmning that we tan choose our base points as described before to be actually fixed points of the group action. We arrive at the following. Lemma A.3. Assume that in the pull-back diagram (46), we have that X, B. and E are ail G-spaces wilh f andp equivariant maps. Then the normalized bar complex (A(.'(),.4c,(),.%(E)) 2792 STEVEN LILLYVHITE is an RCDGA °. Movcover, is a quasi-isomorpbism of RCDGA °'s. If we asu'me further that all fixed-poznt .ets are "on-e'mpty. and the diagram (46) preserves base pohts chosen from the various fixcd-pont sct., the te normdizcd bar complex is at RCDGA. and 0 s a qua.i-isonorphism of RCDGA "s. In this equivariant case, we lnay flnthcr simplify the bar COlnplex, following an idca of [10]. Let us consider the bar COlnplex over R: (57) s, (A5EE (X), AS(S),-«5E(ï)) = ( A5EE(X) c,. (.-5 (S))® «) 5(:). t-0 where ail thc tensor products are over Lemlna A.4. Sqqose tbat ,4.13. avd C are RCDGA "s and we bave .morpbisms of R('DGA "s ,4 -- B - C, wbeve R = H°(BG) for (7, a compact, connected Lie gvow. (II'e use this scquctce to defiw a (differetial gvaded) B-module structure o' A ad C) &ppose, flwtber, eitbev tbat for each r R. r is ot a zevo-divisor ir A. or tbat this conditio't bolds for C. Tbev tbe natural projection (58) t.(A, t. C) t.A, g, c) is a quasi-isomorpbisrn of RCDGA 's. Pro@ We have that Bn(A. 13, C) = Ba(A. 13. C)/V, where 1" is the sub-complex generated by all elements of the form (59) (a. bi ..... rb ..... bk, C) -- (a. bi,..., rb+ ..... ba, c). where r R,a A,b i 13. and c C. It is due to the fact that all elements of R bave even degree that l: is closed under the differential D = d + 5. We clailn that V is, iii fact, acyclic. To see this. consider the lnap s l "i - l-i-1 defined bv (60) s{(o,b I ..... ïl)i,... ,bh.c) -- (a. bl .... ,'rbi+ .... ,ba.c)} = (-1) {(a, bi ..... rbi. 1. bi+l,..., bk, c) - (a. bi, .... bi, r, bi+l ..... bk, c)}, where ai = deg a + deg w +- + deg cri - i. It is straight forward but t edious to check that ris + sd = 0. and that 5s + s5 = id., so that Ds + sD = id., and consequently V is acyclic. Moreover. it is easy to check that l" is an ideal, so that the product on thc bar COlnplex induces a product 011 the bar complex over R. [] Remark A.5. Lemlna A.4 is valid using the normalized bar complex. Corollary A.6. In the situation of Lemma .4.3. (6) O'n(A.(X),A.(B),A.(E))--, Ab(E.f ) is a quasi-isomorpbisn o.f RCDGA "s (RCDGA "s). FOF{IIALIT IN AN EQUI'vAF{IANT SETTING 2793 R EFERENCES 1. C. Allday, Rational homotopy and torus actions, Houston .hmrnal of Math. 5 (1979), 1-19. IlR 80rn:57033 '» , Invariant Sullivan-de Rbam forms on cyclic sers, .!. London Math. Soc. 57 (1998), 478 490. NIR 99g:55015 3. C. Allday and V. Puppe. Cohomological metbods in transformation groups, Cambridge Uni- versity Press, 1993. hIR 94g:55009 4. A. K. Bousfield and V.K.A.M. (;ugenheim, On PL de Rham theory and rational homotopy type, Memoirs of t he American Mathematical Society, no. 179, 1976. MH 54:13906 5. K. T. Chcn, R«duced bar constructions on de Rham complexes, Algebra, Topology, and Ct- egory Theory, pp. 19-32. Academic Press, New York, 1976. hIR 54:1272 6. P. lï)eligne, P. Grilths, .l. Morgan, and D. Sullivan, Real homotopy theory of Kiihler manifolds, Invent. Math. 29 (1975), 245 275. MR 52:3584 7. \V. G. Dwyer and .l. Spalinski, Homotopy theories and model categomes, Handbook of Alge- braic Topology (f.M. Jaunes, ed.), Elsevier Science B.V., 1995, pp. 73-126. MR 96h:55014 8. S. Eileuberg and J. C Moore, Homology and fibrations I. Coalgebras. cotensor product and its derived functors. Cmment. Ma.th. Helvetica 40 (1.q66), 1.q9 236. hlR 34:3579 9. A. Borel, Seminar on transformation groups, Ann. of Math. Sudics, vol..16. Princeton ['ni- versity Press, 19t50. lkIR 22:7129 10. E, Getzlcr, J. lï). S..loues, and S. Petrack, Differential forms on loop spaces and the cyclic bar complex, Topology 30 (1.q91), 339-371. lkI 92i:58179 11. M. Goresky, R. Kottwitz, and R. NIacPherson, Equzvariant cohomology, hoszul duality, and the localization theorem, Invent. Math. 131 (1998), 25-83. MR 99c:55009 12. P.-P. Grivel. Formes différentielles et suites spectrales, Ann. lust. Fourier 29 (1979). 17-37. MR 81b:55041 13. V. Guillemi, and S. Sternberg, Supersymmetry and equivariant de Rham theory, Springer- Verlag, New York, 19.q9. lkIR 2001i:531,10 14. S. Halperin, Lectures on mtnimal models, Méro. Soc. ikIath. France, vols. 9-10. 1983. 85i:55009 15. S. Halperin and .1. Stasheff, Obstructions fo homotopy equivalences, Advances in Mathematics 32 (1979), 233-2ï9. hIR 80j:55016 16. S. Lillywhite, Bar complexes and formality of pull-backs. Preprint. 17. __, The topology of symplectic quotients of loop spaces, Ph.D. thesis, The University of Maryland. College Park, NID.. 1998. 18. __. The topology of the moduli space of arc-length parametrised closed curves in Euclidean space, Topology 39 (2000), 487-494. hlR 2001b:55028 19. G. Lupton, Variations on a con2ecture of Halpemn, Homotopy and Geometry, vol. 45, Banach Center Publications, 1998, pp. 115-135. hlR 99m:55014 20. J. McCleary, User's guide to spectral sequences, Publish or Perish, luc., V'ilmington, DE, 1985. hIR 87f:55014 21. T. J. Millet, On the formality of (k - 1)-connected compact manifolds of dimension less than or equal to 4k - 2. Illinois Journal of Math. 23 (1979), 253-258. hIR 80j:55017 22. D. G. Quillen, Homotopical algebra, Lecture Notes in Mathematics, vol. 43, Springer-Verlag, 1967. hIR 36:6480 23. __, Ratzonal homotopy theory, Ann. of Math. 90 (1969), 205-295. MR 41:2678 24. L. Smith. Homological algebra and the Eilenberg-Moore spectral sequence, Trans. Amer. Mat h. Soc. 129 (1967), 58-93. hIR 35:7337 25. C. Teleman, The quantization conjecture revisited, Ann. of Math 152 (2000), 1-43. 2002d: 14073 DEPARTMENT OF hIATHEMATICS. UNIVERSITh OF TORONTO. 100 ST. GEORGE ST.. TORONTO. ONTARIO, CANADA IISS 3G3 E-mail address: sml@math.toronto, edu TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 355, Nunlber 7, Pages 2795-2812 S 0002-9947(03)03276-8 Article electronicMly published on Nlarch 12, 2003 LARGE RECTANGULAR SEMIGROUPS IN STONE-(ECH COMPACTIFICATIONS NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK ABSTRACT. VP show that large rectangular semigroups caI he round in certain Stone-Cech compactifications. In particular, there are copies of the 2 « x 2 « rectangular semigroup in the smallest idem of (fiN, +), and so, a semigroup consisting of idempotents can be embedded in the smallest idem of (fiN, +) if and only if it is a subsemigroup of the 2 ¢ x 2 ç rectangular semigroup In fact, we show that for an 3" ordinal ,k with cardina.lity at most ¢, fin contains a semigroup of idempotents whose rectangular components are ail copies of the 2 « x 2 ¢ rectangular semigroup and form a decreasing chain indexed by ,k + 1, with the minimmn component contained in the smallest ideal of fiN. As a fortuitous corollary we obtain the fact that there are <L-chains of idempotents of length ¢ in fiN. \Ve show also that there are copies of the direct product of the 2 c × 2 c rectangular semigroup with the free group on 2 ç generators contained in the smallest idem of fiN. 1. INTRODUCTION The Stone-Çech «ompactification of the integers fin has a semigroup structure which extends addition on N a.nd has significant applications in Ramsey Theory and topological dynamics. Some questions about the algebra of fiN, which sound deceptively simple, have been found to be extremelv ditïtïcult. For example, it is hot known whether/3N contains any finite semigroups whose members are hot all idempotent. VChether there were two idelnpotents in /31%1 whose sure was an idempotent different Kom either remained an open question for several vears. It was answered in the aflïrmative in [101, in which it was shown that a certain finite rectangular semigroup could be embedded in/31%1. (A semigroup is rectangular if and onlv if it is isomorphic to the direct product of a left zero semigroup and a right zero semigroup. A rectangular compo'net of a semigroup of idempotent, is a nmximal rectangular subsemigroup. As suggested bv the naine, distinct components are disjoint. The components are partially ordered bv the relation /9 _< Q if and only if/9 0 C /9, equivalently 0/9 Ç /9 [7. Theorem 1].) In this paper, we show that the rectangular semigroup 2 « x 2 with the first factor being left zero and the second right zero. can be embedded in /31%1. Indeed. pin contains semigroups of idempotents which are the union of ¢ rectangulai components each isomorphic to 2 « x 2 «. We shall show also that if S is an infinite cancellative semigroup with cardinalitv e:, then S \ S contains a semigroup of idempotents which is the UlliOll Received by the editors April 12, 2002 and, in revised form, November 14. 2002. 2000 Mathematscs Sub3ect Classification. Primary 20Ml0; Secondary 22A15, 54H13. The first author acknowledges support received from the National Science Foundation (USA) via grant DMS-0070593. @2003 American lklathematical Societ 3 2796 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK of at least t rectangular COml)onents, each isomorphic to 2 2" x 2 , with the first factor being lefl, zero and the second right zero. X first review terminology used in the topological theory of semigroups. Let ,q' 1)e a semigr¢mp and a topological space. For each s .', we define mappings A. and p. from S to itself l»v A,(/) = .«t and p(t) = ts. ,ç is said to be a right lopologicol semigroup if p. is contimous for every s S. In this case, the topological tenter A(S) of S is defined by A(S) = {s S A is continu(nlS}. S is said to l)e a ,semitot)ological semigrou I) if ils and As m'e hoth contimous %r every s G ,ç. It is sai(l te) t)e a topologicol scmigroup if the semigroup Ol)eration is a continuous lnal)ping from .%' X ,5' to If ,S' is a discrete semigronp, we regard its Stone-'ech «ompa«tification 5' as the spaçc of ultrafilters defined on ,5 . with the topology defined bv choosing the sers of the fonn 4 = {p G/35 .4 G 1»} as a baso for the open sers. t3S is then a compact tIausdorff simce and . = ('ÇBs(4). XX]' regard S as a subset of 3S, b identifying oach clé,mont of .h' with the principal ultrafiltor that it defines. N can he given a scmigroup structure which extonds the scmigronp structure of S in snch a way that b' is a compact right topologiçal scmigronp, with ,S" contained in its topological ('enter. If A ç S..I* will denotc A A. XX3 shall nso basiç algehraiç 1)r»i)crtios that hold in all compact Hansdorff right topologi«al scmigroups. (XX shall ho assmning that ail hypothesized topological spaces are ttausdorff.) A simple and important property is that everv compact right topological semigroup ç contains an idemt)otent. T has a smallest ideal K(T), which is hoth the union of ail the lnillimal left ideals and the union of all the nlillinlal right ideals of T. Every right idem of T contains a lnillilnal right ideal. and every lefl ideal of T contains a mininlal left ideal. If L is a minimal lefl ideal and B a lnininlal right ideal in T. tllen RL = R L is a group. So B L contains a unique idempotent. If f T T' is a honlolnorphisni ff'oin T onto a COlnpact right topologi«al grou I) T'. thon f[K(T)] = K(T'). For each minimal right ideal R' of T', there is a minimal right ideal R of T for which f[R] = R'. The corresponding statement holds for left ideals as well. There are three natural orderings of the idempotents of T defined C<_Lf <= e = cf , e<_Rf e= fa. and c<_f of=fa=c. An idempotent e is minimal with respect to any or ail of these orderings if and onlv if e e K(T). The reader is referrcd to [1], [6], or [9] for proofs of these statements. When S is a discrete semigroup, the smallest ideal K(/3S) is of special importance for combinatorial applications, and in particular, the members of idempotents in K(/3S) have strong combinatorial properties. (Sec [6, Chapter 14].) Thus we are especially interested in those scmigroups of idempotents that tan be embedded in the smallcst ideal of/_S. As we have already mentioned, a semigroup S is rectangular provided it is iso- morphic to the direct product of a left zero semigroup with a right zero semigroup. This is equivalcnt to saying that it satisfies the identities .r 2 = x and .ryz = xz. (The necessity is trivial. For the sufficiency, pick x S, note that Sx is a lefl zero scnligroup, xS is a right zero semigroup, and the fimction (a. b) - ab from S.r x xS to ,_q is ml isonlorphisnl.) We observe that a rectangular semigroup S FIECTANGULA1R. SEMIGIROUPS IN STONE-CECH COMPACTIFICATIONS 2797 satisfies S = N(S) = LR L x R. where L deuotes anv minimal lefl idem and R any nlinilnal right idem in S. If S is a senligroup, E(S) will denote the set of idempotents iu S. If A is anv set, 72I(A ) will denote the set of finite nonempty subsets of A. '2. TtlE SEMIGIOItPS The subsemigronp ]HI = cg(N2 '') of (tiN, +) hokls ail of the idempotents L of ÇN and lllllc of the klloWll algebraic structure. (Sec [6, Section 6.1].) It oecllrs widelv in the stndv of semigroups of the fi»rm /3,% If S is an infinite discrete cancellative semigroup, every G6 subset of S* that «ontains an idempotent, contains copies of N [6, Theoreln 6.32]. N also bas the property that anv compact right topological semigroup with countable dense topological eenter is the ilnage of uuder a contimmus honomorphism [6. Theorem 6.4]. In this section we introduce a senligroup . whi«h satisfies a similar conclusion fin" an arbitrarv infinite cardinal . As a consequence of the results of thc next section we shall conchule that ea«h N. contains large re«tangular subsemigroups. Definition 2.1. Let h be an infinite cardinal. Thon supp(z) : {t < n " .ca ¢ 0}. For et < n, eu is that nlember of II such that supp(ea) = { a }, and The structure of is that induced bv an "'oid'" as imroduçed bv John Pym [8]. When we sav that two structures are "topologically and algebraically isomorphic". we nlean tllat there is one fim«tion hetween them that is both an isomorphism and a honeomorphism. Theorem 2.2. The compact right topological semigroups attd are topologi- call9 and algebraicall isomorphic. Pro@ [6, Theorem 6.15]. It is a fact [6. Lemma 6.8] that ail of the idempotents of J3N are in IHI. Thus, bv [6, Theorem 1.651, Iç(]HI)= Iç(3N) a IHI. Theorem 2.3. Let S be a countablg mfinite discrete gT"oup. Then ris \ S contains a topologi«al and algebrai« copy T of IHI such that Iç(T) = Iç(/3S) ç T. Pro@ Take any idempotent p C K(/3S). By [6, Theorem 9.13], there is a left invari- ant zero-dimensional Hausdorff t.opology on S in which the ultrafilter p converges to 1. Then by [6. Theorem 7.24], with X = G = V(a) = S for everv a C G, there is a topological and algebraic embedding f : IHI --* ris \ b' such that p f[IHI]. It remains to apply [6, Theorem 1.65]. [] A similar result applies to the senligroup (N,-). Given n G w we define the binary support of n by n = Etesupp2(n ) 2 t and supp2(O Theorem 2.4. Let S = (N, .). There is a topological and algebraic copg T of contained in /3S\S which contains all of the idempotents of S\S. In particular. 2798 NEIL HINDMAN, DONA ST1RAUSS, AND YEVHEN ZELENYUK Pro@ Let (pi)ici__ 1 be the sequen«e of primes. Then (Oi-----1 03, --) is isolnorphic to (N, .) via the lllftlï) let tri : S co be the projection to the ith factor and let i : /3S -- /3o be its contilmous extension. Let {Xi : i N} be a partition of o into infilfite sers and for each i IN, let c)i : Xi -- o be a bijection. We define 0 : co -- S by agreeing that for each i N and each n where ïje 2 e'(j) = 0. k ' note that 0 is a bijection. (If j supp(n) supp(m), then for Solne i, j G Xi and so i(j) sui,P2 ((n)) supp 2 (i(m)). Also, given z S. fbr each i N let = i-l[supp (i(x))], let Z = ie }, and let ' = tez 2t" Then 0(n) = .r.) Colsequently by [6. Exercise 3.4.1] the continuons extension S of 0 is a bijection. SillCe 0(n + m) = 0(n) +0(m) if supp2(n ) supl)2(Ttt) = , is a holnOlnorphism on 1,y [6, Lelmna 6.3]. lb çolnplete the proof, let p be an idempotent iii SS. Since a[NI= [N,=ll{0}: N,,=0[]k{0}. it suces to show that for all n N, 0[2"] p. So let n N and suppose that 0[w2 ' ] p. Pick t e {1,2 ..... 2'- 1} su«h that 0[w2 + t] e p. pick j e supp2(t), and pick i such that j Xi. Now Yi(P) is an idempotent: so either (p) = 0 or (p) flNN. Tlms by [6, Lelnlna 6.6] w2 e'(j)+l p. Pick z w2 ()+1 N i[O[2'+t]] and pi«k k e 2"+t such that .r= i(O(k)). Then j e supp2(k)NXi; so i(j) stt)P2(x), contradicting the fact that z 2 (j)+l. Observe that if > w, then N N K(ll ) = . To see this, one lets p N and q e Nk<w d {e IlÇ{} nlill sllpp(w) 2 and lllax supp(z) < w'}. Then p I1 + q + p and so [6, Theoreln 4.39] applies. Theorem 2.5. Let be an inflnite cardinal and let T be a compact right topological semigro,p. Assume that there is a set A ç A(T) s,ch that lAI and A is dense in T. Then there is a contim, o,s s,wjective homomorphism f " N T. Pro@ Elmlnerale A as {t : a < }. with repetition if lAI < . Let {I : < } be a partition of into subsets of size . Define h IV + T by first agreeing that for each o. < t;, h(e,) = tx, where o. I. h(_.a F ea) = HaF h(ea), where the product is indices. Define h(0) arbitrarily. Let ï :/3II, -- T be and let f be the restriction of to IE,. Then for F 7)(t), define taken iii increasing order of the contimous extension of h To see that h [IE,] = T, it suffices to show that A C_ h[lE,]. Given "7 < t, we have that II[ = t. Pick a t-uniform ultrafilter p on {ec a I-}. Then p IE, and h(p) = tv because f is constantly equal to t- on {ec a L}. To see that f is a homolnorphisln it suffices by [6, Theorem 4.21] to observe that whenever a: IIÇ\{} and / IlÇ\{} with rein supp(/) > max supp(x), then (z + v) = h(z). (v). [] Definition 2.6. Let S be a selnigroup, let h; be a cardinal, and let (t,x),x< be a -sequence iii S. (a) Given F 7)l(n ). 1-leF t is the product in increasing order of indices. RECTANGULAR SEMIGROUPS IN STONE-CECH COMPACTIFICATIONS 2799 (b) If D C_ h;, then FP((t)aD) = {IIF ta F E PI(D)}. (c) The sequcnce (ta)a< has distinct finite prodacts if and onlv if whenever N. G E PI(A) and 1-[,XeF t,x = 1-[aec fa, one must bave F = G. Theorem 2.7. Let S be an infinite cacdlative discrete se'migroup with «ardmality ris \ S coutoins a topological ad algebraic copy of ,. Pro@ By [6, Lemma 6.31], we nlay choose a t-sequen('e (ta)a<,, in S with distinct finite products. Let T = FP((t)x<). For each V < , let Tv = FP((t),<x<). U put ç = < «s(T). By [6. Theorem 4.20], is a subsemigroup of riS. define 0 T I1 by 0(xe tx) = xe «» (Since the sequence (t)a< has distinct finite products, the fimction 0 is well defined.) Let clos(T) flllç denote the continuous extension of 0. By [6, Theorem 4.21], the restriction of g to ç is a homomorphism. Nov is injective, by [6. Eercise 3.4.1]. Since for each 7 < , [T] = cft3u-{x e II/ lnil SUl)p(x) > 7}, maps onto N. Thus determines ail isomori)hisnl from OllIO . 3. CHAINS OF RE('TANGI'LAR SEMI(IROI'PS 1N - Let be an infinite cardinal and let IÇ denote the rcctangular semigroup with the first factor 1)eing left zero and the second right zero. We show in this section (in Crollary 3.10) that for mly infinite cardinal h;. algebraic copies of can be round in tç(NI,). Indeed. if A is any ordinal for which I)q _< , there is a (lecreasing chain (D»)»<a of diioint copies of I.), contained in NI, with Da embedded in tç(NI,). Notice that 1/ contains a copv of cvery l'ectangular semigroup of cardinalitv at lllOSt Definition 3.1. Let A be any ordinal and let A be all.V llOllelllpt.v set. Let 0 denote a selected elelllent of A. For p < A. let Cp = A x 4 x {p} and let C = Ca,a = A U Up<aCp = A U (.4 x A x A). The operation on C is defined as follows. Let a, b. c, d E .4 and let p. q < A. Then a.b = b, o. (b,c,p) = (b.c,p), (b,c,p).a = (b,a.p), (a,b,p).(c,d,q) = (a,d, pVq), where p V q is the maximum of p and q. V'e leave to the reader the routine verification that the operation on CA.a is associative. Notice that for any p < A, Cp is a cop.v of l/iA I. Definition 3.2. Let A be an ordinal and let p < A. let vp = (0,0.p), and for every x = (a, b.p) e C, we let xt = (a. 0.0) and Xr = b. For x A, we let The folloxving is silnple, and its proof is like that of [6, Theol'en 1.46]. Lemma 3.3. Let S be a semigroup, let H be an ideal of S. let L be a minimal l@ ideal of H. let be a minimal right ideal of H, and let z E S. Then Lx is a minimal l@ ideal of H. .rR is a miimal right ideal of H. xL ç L. and Rx ç . 2800 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK Pro@ It suffices to establish the assertions about La" and 3"L. Now L.r C H.r C H and HLx C L.r: so Lx is a left kleal of H. Let M be a lcft kleal of H with M ç Lx. Let.l={gL'gxM}. GivcngJandzH, wehavezyLand zyx3l: so zg Ç J. Thus .] = L and so M = L.«. Next. givcn g G L = Hg, so pick z Ç H such that y= zy. Then xy=xzy Ç H9 = L. Lemma 3.4. Let A bc o nonempty set with di.çling«ished element O. and let C = C.4,1. Let T bc a right lopological semigroup, and let f T C be a surjective homomorphism for u,hich f-[A] and f-l[Co] are conpact. Then there is a homo- morphism g C T sach that f o g is the idcnlity on C and 9[Co] ç Iç(f-[Co]). (f T is compoct, then 9[Co] ç K(T). Ptv@ V\, first (lvfine g on .4. We have that f-1 [.4] i8 a COlnpa('t semigroup. Choose a minimal right idem Nof f-[A]. For ea«h o A. f-il{c,}] i . ,,f id(, of f-l[A]. So (.hoose a llininal l(,ft id«al « of f-l[A] with S ç f-[{}], ald let g(a) be the identity of the gloup N a- Then immcdiately f(g(a)) = a. Also. given a, b Ç A we have that .q(a) and g(b) are idt'mpotents in N: so g(a)g(b) = g(b) = g(ab). Let = {(,.o.o):o 4}. Tll('ll is a l('ft idçal of Ç0: so f-l[B isa left ideal ,,f f-a[c0] whi«h thelef(n'e contains a minilnal left ideal L of f-l[C0]. For each a G A let F = {(o. b, 0) : b G A}. Then F is a right ideal of C0. So pick a lnininml right ideal R of f-[C0] with R ç f-lirai. Bv LCIlIIlI& 3.3, SillC f-l[C0] is ail i(leal of f-l[A uÇ0], we bave that 9(0)- R is a nfiniliml right ideal of/-l[C0] and L. 9(a) is a nlinilnal lcft ideal of f-z[C0]. For a. b G .4. let 9(a, b. 0) be the identity of the group 9(0)" R. L.9(b). Notice that if T is COlnpact. then K(T) ç f-[C0] so that Iç(f-l[C0]) Iç(T) and thus 9(a, b, O) K(r). Also 9(a, b. O) = 9(O).x.-g(b) for SOllle .l'e a alld SOllle e L. So f(.q(a,b.O)) = O. f(x). f(y).b= (a.b, 0). To conclu(le the proof we nee(t to show that g is a hOlnOlnorphisln. First we let o. b, c Ç A and show that g(o) .g(b. c. O) = g(b. c. 0) and g(b. c. O) .a = g(b. a. 0). Pick .r H6 L such that g(b. c, O) = 9(0) x. g(c). Then g(o).g(b,«.O) = (,)-9(0)-.,-9() = 9(0)- .- = 9(b, «, 0). So the first clainl holds directlv. Multil)lying on the left by g(b.c. 0) and on the right by g(a) Olm sees that g(b. c. 0)-g(a) is idelnl)otent. Since g(b.c. 0). g(a) 9(0)" Rb" L. 9(c). g(a) = 9(0). Rb" L. 9(a), we nmst bave that g(b. c, 0). g(a) is the identity of g(0) Rb" L. 9(a), nalnely 9(b, a, 0). Finally, let a, b. c, d Ç 4. Then 9(a. b. O) 9(c, d, O) Ç 9(0) R, L g(b) 9(0) R«.L.g(d) c 9(O)-R«-L'g(d). So it suffices to show that g(a.b.O).g(c,d,O) is idempotent. These elements satisfy 9(a.O.O) L g(O) and 9(c, 0,0) L 9(0). So as idelnpotents in the saine lninimal left ideal of f-[Co], we bave that g(a, O, O) 9(c, 0.0) = 9(a. 0.0). Recall that we bave shown that for any x. y A. 9(0)-g(x,y,O) = g(x,y,O) and g(.T.y,O)-g(O) = g(x,O.O). Thus we have g(a, b, 0). g(c. d. 0). g(a, b. O) = 9(a,b.O).g(O)-g(c,d,O).g(O).g(a.b.O) = g(a,O.O).g(c.O.O).g(o,b.O) = g(o,O.O).g(a,b,O) = g(o,b.O).g(O).g(a.b.O) = 9(a,b.O). g(a.b,O) = g(a,b, 0). IECTANGULAIR SEMIGROUPS IN STONE-CECH COMPACTIFICATIONS 2801 Multiplying on the right by g(c, d, 0) we have that g(tt, b. 0).g(c, d, 0) is idempotcnt. We now cousider the situatiou in whi«h k > 1. For k > c0 we do uot necessarily get that g is a homomorphism, hut xve corne Theorem 3.5. Let A be a nonenq#.q set u,ith distinguished element O. let A be an ordinal, and let C = Ç4.. Let T be a right topological semigmu p, and f : T be a su«jectiue t, omomo,vli.m .sud, tlmt f-[A] i.s. compact and f-l[c] is comp,wt for every p < A. The'n there is a function g : C T su«h th«.t f o g is the identity attd 9 bas the .followin 9 pvperties: (i) If q p < . G Cp, and g G AUC. then 9(«g) = g(a')'9(g) and g(g).g(.r) is art idempotent in thc saint minimal lefl id«al 4 f-'[G] C,,s 9(.qx). (ii) lf p < A. « G C v. and U G 4 Co, then 9(g) " 9(.v) = 9(g.r). (iii) ff q p < A. n G . p = q+n. g G C. and x (iv) gp < A. ther, 9Cp] lç(f-l[cv]). The seigrou p T «ontins semigrou p D = p«A Dp of idempotents where ea«h is a ,ecta,gthtr component of D with g[Cp] Ç Dp and the scquence {Dp)p< is de- cre«sing in the orderin 9 of components, so that for each p < 6lAI IX] w. then for et, ch p < X. Dp] i.ç i.çomo,phi« fo I]A I. Proof For p < X we &'fine g on A U qp Cq 10 induction on p so that g satisfies conclusions (i), (ii), (iii), and (iv). By Lemma 3. we mav definc 9 on A U C0 so that g satisfies (iv) and is a homomorphism and therefiwe satisfies (i). (il), and (iii). Now let (l < p < X and assume that 9 has been defined on 4 U q<p Cq. IO show fiist that we moEv choose a minimal left idem L of f-[Cp] such that L ç q<vf-l[Cp].g(%)and fiL] = Cp-%. A si,uple computation establishes that x. 9(q) = .. g(), g(,,,.) f-'[%]. (,,). by (i) Consequently. r-]q<p f-'[Cpl- g(uq) is a lefe ideal of f-l[Cp] mM thus contains a mi,,i,nal left idem L of Now giveu .r L, one bas x f-a[G] g(uo). So for sou,e «,. A. f(x) = (a,O,p) G- up. Thus f[L] If p is a successor ordinal, observe that g(%_).f-[Cv] is a right id « f-[Gl and pick a minimal right ida is a minimal right ida « G ad f[l ç %-" G: o f[l = -1" G- If p is a limit ordinal, note that f-[%- Cp] is a right idem of f-[cp]. So pick a minimal right ideal Now f-[cp] is an idem of f-[AuCUCp]. So by Lemma 3.3. for any x Cp, g(xt) - R is a minimal right idem of f- [%] and L-g(r) is a minimal lefl idem of f-l[c A. çhr«, (Ot)" g(a') be the identitv of 0(.re) R- L- O(a>). Notice that (iv) is satisfied. 2802 NEIL H[NDMAN, DONA ST1RAUSS. AND EVHEN ZELENYUK To verify (i), let q < p. let x Cp, and let y A tO Cq. To see that g(xy) = g(x). g(y}, we show that g(x} g(g) is an idempotent in the saine grouI) as g(xy). Since (J'g)e = xe and by Lemma 3.3, R. g(g) Ç R. we have that g(xg) g((xge)) " R = (-«) n ,« g(,). () (e) n-() ç (e) n. also. (a.g},. = y and so g(a'g) L.g((xy}) = L.g(y). ço see that g(x).g(y) e L. g(y), we consider two cases. If g e Q, then g(x) g(g) e g(x) L. g(Yr) ç L" g(y,.). Now assmne that q < p (and y A U Cq). Note that L ç f-[Cp]. g(«q); so g(uq) is a right identity for L and thus L : L. g(uq). Also. a simple computation establishes that uuy : uqy. Therefore, using the fact that (i) holds at q, g(uq) . g(x) . g(y) : g(«qx) . g(y) = g(t, qxy) = g(uqy) : g(uq) . g(y) and th, g().9() r.9().9(y)= r-9(,)-9(«)-9(Y)= r'9(,)9(>)= 'g(>)- (.onsequ¢ ntlv, we have in anv event that g(xy) and g(x). g(g) are members of the group g(xt) R . L. g(y). show that they are equal bv showing that g(x) g(y) is idempotent. Since g(x), g(0) : g(.r,.0) = g({}) : g(g0) = g(y) g(O) we have that g(x) - g(O) L. g(x) g(0) = L-g(0) and be«ause g(x) g(y) L. g(y) we bave that g(«). g(). 9(0) . g>). 9(0) = L. g(0). w g() = 9(.,c). -9() o- ,, z -n..o 9(0)- 9(*e) = g(0e) = 9(xe) and so g(0) g(x) : g(xe) . z. g(x) : g(x). If y G Cpwe bave similarly that g(O). g(y) : g(y), while otherwise g(0)- g(y) : g(Oy) : g(y) by (ii) of the induction hypothesis. have that g(x)-g(0)-g(x), g(O) : g(x) . g(x) . g(O) = g(x) . g(O). So g(x)-g(0) is an idempotent in L. g(O), which is a minimal left ideal of f-l[cp by Lenmm 3.3. Therefore g(x) g(0) is a right identity for L g(0) and thus g(x) 9(). g(0). g(z). g(0) = 9(-). 9(). 9(0). So g(x). g(y). g(x). (y) = v(,). v(y). g(0). v(,), g(0). = g(x). v(). v(0). v(v) = (,). g(y)-(y) = («). as required. Bv Lemma a.a. oy).g(xe). R ç f-l[cp]; so (y)'g(x) E L-g(«,.). Also, g(yx) e L. g((gx),) = L. g(x,) and by Lemma a.a. is a minimal left ideal of f-l[Q,]. To see that g(y) g(x) is idempotent, note that xyx = x. So g(x). g(y). g(x) = g(xg)- g(x) = g(.'gx) = g(x) and t5us g(g) . g(x) . g(y) . g(x} = g(g) . g(x), as required. This completes the verification of (i). Toverify (il), let. x C, and let y AtoC0. Pick z L-Rsuch that g{x) = g(xe) z-g(x,). Then g(y) g(x«) = g(yxe). If y -1, then yxe = xe so that g(y).g(x) = g(y).g(xg).z.g(x,) = g(xg).z-g(x,) = g(:r} = g(yx). So assume that g E C0. Then yxe = (yx)e and (y«), = x,. So g(y). g(x) = g(y). g(xe)" z. g(x,,) = .q(yxe). z.g(x,) --- g((yx)t), z.g( (yx),). So to see tiret g(y).g(x) = g(yx) it suffices to recall ïrom (i) that g(y) g(x) is idempotent. To veriïv (iii), let n w and let q <_ p such that p = q + n. Let x Cp and let g ff Cv. If n = 0, the conclusion ïollows ïrom (i). So assume that t > 0 so that p is a suc«essor ordinal and p- 1 >_ q. Now (ya')e = ye: so g(yx) ff g((yx)g). R = g(gê).R. IRECTANGULAR SEMIGIROUPS IN STONE-CECH COMPACTIFICATIONS 2803 Recall/hat R C_ 9(up-) f-x[C v] and conse<lUently R = 9(up-,)" . Thus, 9(9)-9() 9()9(,r«)- = g()'g(,r«)'g(llp_l)" = g(9.rc)-g(v-)" R by (i) at q = g(,rftt, p_l). R by (iii) a.t p- l = 9(yeup-)" R = g(Ye) "g(up-1) R by (ii) at p- 1 = g(e) . Now 9(Y)" 9(x) is an idempotcnt in thc sa.me minimal left ideal of f-[Cp] by (i). Since g(yx) and 9(Y) " g(x) arc also in thc saine minimal right ideal g(y() t. they must be equal. This complotes the induction step. Next, we establish (v). So assume that T is compact and A is a successor. Thon Iç(T) Ç f-l[C_l] and so h-(f-l'[C_]) ç h'(T). Sin«e g(x) e lÇ(f-l[CA_l]) for every x C-l, (v) holds. 5r each p < A, let = { (') F e ç«(a), » = ,,,x F, ,,a fr «h q e F, ' e C}, where for each F, thç product qF g(,rq) is taken in increasing order of indices. show now bv induction on IF] that ifp < A. y Cp. F çy(A), xq Cq for each q F, (*) and maxF p. tlwn g()'qF g(q) = g(Y'qF q)" Let r = maxF. If F = {r}, then g(y)-g(x) = g(yx) by (i). So assume that () « (.) = () a (Æ)a() = 9(Y" ea xq) g(x.r) by the induçtiou hypothesis = a(a r) b (i). Now we show that each member of D is idempotent. So let F çy(A). let p = nmx F, and for each q F, let xq Cq. If F = {p}. then g(Xp) is idempotent. So assume that IFI > 1 and let a = F {p}. Then « a() « () = a a(«) a(«) « () = a («) = a ()a( = a ()a() = qF g(.rq) . Now let r.p < A, let a Dr, and let b have that a = HqeF g(xq) and b = Hqea g(Yq) where maxF = r. maxG = p. each xq e Cq and each yq e Cq. If r < llin a, then ab = qeF g(zq) . qea g(Yq) e Dp. Ifr k p, thenab = qeFX{-r} g(xq)'g(xr'qG yq)e Dr (where the qeFX{} g(xq) term is simply omitted if F = {r}). So assume that rein G G " q r}, and let L = {q e G g > r}. Then ab = HqeF{} g(xq).g(x. ,u). (u) e D. Thus D is a semigroup of idempotents and for each p < A, K(Uq p Dq) Dp. Let r p, let o = qeF g(xq) e Dp, let b = qG g(Yq) Dr, and let c = 2804 NEIL HINDMAN, DONA STRAUSS, AND YE¥ HEN ZELENYUK H,eH g(zq) Dp. Then abc = H qF\{p} = II qF\{p} qeF\{p} qG qH (.1. (» . H .q " H z) qG qH g(.rq)- g(XpZp) alld (c - H g(.rq)-g(.rp- H qF\{p} qH qeF\{p} So abc = ac and so each Dp is a rectangular subsemigroup of D. To see that D v is a rectangular component of D, suppose that a Dp and b Dq, where q < p. Then f(bob) C and f(b) Cq, and so b,b ¢ b. To sho« that Dp IiA I if lAI > [,Xl > a, we observe that Cp contains a left ideal L and a right ideal R, each with 1,4[ elements. If a,b L, then ab = a and so g(a)g(b) = g(a). Thus g[L] is contained in the left ideal Dpg(b) of Dp. Silnilarlv. g[R] is contained in a right ideal of Dp. So Dp contains a left ideal and a right idem each with at least elements. They cannot have more than lAI elements because for each F with p = lnaxF, there are lAI IFI = lAI choices for 1-[qeF g(Xq). So [Dp[ = 141. Thus D v , L x R . IAI- [] Two ol»vious questions are raised by Lelnlna 3.4 and Theoreln 3.5. First, can the flmction 9 constructed there be required to be continuous? Second. tan the flmction 9 in Theoreln 3.5 be required to be a honollorphism? We shall answer bot h of these questions in t he negative, even w hen the stronger requirements that T and (7 be compact and (7 be a topological semigroup are added. We shall ha:ce lmed of the following lemma, whose routine proof we omit. (Recall that anv successor ordinal is a COlnpaçt Hausdorff space under its order topology.) Lemma 3.6. Let .4 be a compact space, let A be a ordinal, let A x A x (A+ 1) bave the product topology, avd let A and A x A x (A + 1) be clopen subsets of C = CA.X. Thev (: is a compact topological semigroup and (' = K(C). We now show that. eVell for A = 0, one cannot require that g be continuous. "Ve remind the reader that an F-space is a completely regular space A" in which {x X : f(x) > 0} and {x X : f(x) < 0} are completely separated for ail continuous f : X Theoreln 3.7. There eïist a nonempty set A. a topology o'n C = CA, 1 8uch that C is a compact topological semigroup and A and Co are compact subsets of C. a compact right topological semigwup T. and a continuous surjective homomorphism f : T -- C such that there is no cottinuous homomorphism g : 6' --} T for which f o gis the identity on C. (In fact. there is no cotinu.ous injective function from C roT.) RECTANGULAR SEMIGROUPS IN STONE-CECH COMPACTIFICATIONS 2805 Pro@ Let A = pIN, let C = fin U (fin x fin x {0}) with the topology given in Lelnlna 3.6, and let T = H. Then N U (N x N x {0}) is dense in C = A(C). So there is a continuous surjective homolnorphism f T Ç by Theorem 2.5. Now suppose there is a continuous injective flm«tion 9 " T. Then by Theorem 2.2 there is a continuous injective flmction ff'oin Ç to ç fiN. But this is ilnpossible because fin is an F-space [3, Theorem 14.25]. So evel'v compact subset X of/IN is an F-space, because everv COlltillUOllS function fronl .ç to [0,1] ll a continuous extension to fiN, by the Tietze extension th(,oroln. But fin x fin is llOt an F-space by [3. 14Q]. Theorenl 3.8. There exisl a nonempt set A u,ith distinquished el«ment 11, a topol- o9 on C = CA,+I suc tat C is a compact topological semigrou p aud A are compact su.bsets of Ç for each p ', a compact rtgt topological semigroup T, and a continuous surjective homomorpisnt f " T C suc tot lhere is no homomotph.ism 9 " C T for whi«h f o 9 is the identit.q o ('. Pro@ Let A = {0} raid let C = CA.+a with the tolmlogy given in Lemma 3.6. Let u0 = 0. forp < w, let up+a = (O.O.p), and let u = (0.0. w). Th(,n C = {p "p N w} 811d Upq = llpv q fi)l" ail p, q w. "[blologically, u is the onlv non-isolated point in C Let (vp)v< be a sequence of distinct points none of which are in C. Let T = {u v p < ,} U {vp p < «} and for p.q < ' define an operation on T as follows: IIpq IIp'q l'pq Up'q p . b leave it to the reader to verifv that the operation is associative. Let ç {e0} be discrete and let T be the one point compactification of T {v0} (with t'0 the point at infinity). claire that T is a riht topological semigroup. Let p < . To ste that p is contimous at v0. let Il be a lmighborhood of '0 = (0) .d t U = U'({" To see that p, is contimous at v0. let II be a neighborhood of v0 = P,(0) and t u = {% . p q < ..d Define f" T Ç by f(up) = up and f(vp) =u. for eachp < w. Then f is a continuous surjective honlomorphisln. Suppose that g C ç is a honlomorphisnl for which fog is the identity. Then for p < such that g(u) = vq. But then. vq+ = g(u) = eq. a contradiction. shall see next that we can get the function 9 to be a honomorphism bv requiring that T be Selnitopological. This corollary tan then be viewed as saying that Ç is something like an absolute co-retract in the category of semitopological semigroups. Ç becomes an absolute co-retract in the category of compact semi- topological senfigroups if it is given a topology for which it is in this category with A and each Çp being compact. Corollary 3.9. Let A be a nonempty set wilh distingished elemenl O. let A be an ordinal, and let Ç = Ç.4.a. Let T be a senitopological semigroup, and f T Cbe a continuous homomorphism such that f-'[A] is compact and f-[Cp] is compact for every p < A. Then ther« is a homomorphism g ' C T such that f o g is the identity. 2806 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK Proof. At stage p of the induction in the proof of Theorem 3.5 one has that for each q < p, g(Uv)- f-l[cp] is a compact right ideal of f-[c]. So one may choose a minimal right ideal R of f-l[cp] with R Ç q< g(uq), f-[%] and f[R] Then. if G Cq for some q G p and G Cv, just s one showed in the verification of hypothesis (i) that 9(x) 9(P) L- 9(g), one can show that 9(9) "9(w) 9(9t) " R, so that 9() " 9(x) and 9(9x) are idempotents in the saine group. If 9 A and x Cp, then 9(U)" 9(x) = 9(u.r) by (ii). ( did hot need to consider the case y G A separately at that point in the proof of Theorem 3.5 because the equation qr = tlqr '-q valid in any event. The corresponding equation y.reUq = ç(Uq is hot valid if y A.) now prescrit some immediate consequences of Theorem 3.5, although with a bit more effort, wc shall get a stronger result, namely Theorem 3.16. Corollary 3.10. Let be an ifinite cardinal and let A be an ordinal with Then cotains a subsenigroup D = UpA Dp of idenpotents wher¢ each Dp a rectangular comportent of D isomorphic fo I ad the seqence (Dp)A is de- çasing i the ord('ring of çompoent. o that for ea«h p A. Dp = K(UqS Dq). Proof. Let a have the discrete topology and let 4 = fla-. Let C= 4.+ and let lmve the topology describcd in Lemma 3.6. Let T = . Since U( x x (A+ 1)) is a dense sui»set of C = A(C), by Theorem 2.5 there is a continuous surjective homomorphism f T C and so Theorem 3.5 applies. Corollary 3.11. Let ,S' be n infinite cancellative discr¢te semigroup with cardi- ality ad let A be an ordi,al with lAI G a. Then ils S contabs a subsemigrop D = UpA Dp of id¢mpotents where ¢ch D is a r¢ctngdar coponent of D isomovphic fo l/ and the seqence (Dp)pA is decreasing i the ord¢ring of com- o¢t., o t.t fo ¢« p . Dç = ç(U; G)- If s = (N, +), s = (N,-), o S is coudabl nfinite discrete gmp. then D ç K(flS). Proof. By Theorem 2.7, ils S contains a topological and Mgebraic copy T of . (If S = (N, +), choose T = . If S = (N,-) or S is a countably infinite discrete group, choose T in Theorem 2.4 or Theorem 2.3 respectively.) Then Corollary 3.10 applies. If S = (N,+), S = (N,.), or S is a countably infinite discrete group, then K(T) = K(çS) aT So bv Theorem 3.5, with A+ 1 in place of A. we bave [c] ç K(ç) ç(S) ana so D ç K(S). Corollary 3.12. Let S be a contabl infinite discute grop. Then there is a copy of ç« contained in K(flS). K can completely clmracterise the semigroups of idempotents that can be cm- bedded in K(flN). Corollary 3.13. nality t and let D (i) There is a (ii) There is a Proof. Conclusion 3.11. Assulne now [D I _< I/3NI = 2'. Let S be an infinite cancellative discrete semigroup with cardi- be a semigroup of idempotents. copy of D in ris \ 5? if D is rectangular and lOI < 2 2" . copp of D in K(flN) if and only if D is vectangular and [D[ < 2'. (i) and the sufficiency of (ii) follow immediatelv from Corollary that D is a semigroup of idempotents contained in K(flN). Then Next observe that any subsemigroup of idempotents in K(flS) IRECTANGULAIR SEMIG1ROUPS IN STONE-ECH COMPACTIFICATIONS 2807 must be rectangular. To sec this, sui)pose that a', y, z E K(13ç). Then zz and r9z belong to t.he saine minilnal left ideal and t.o t.he saine nlinimal right, ideal. Hence, if they are i(telnpotent, they lnUSt, be equal. Recall that any two nlaXilnal groups in the slna.llest ideal of a COlnpact right topological semigroup are isomorI)hic. çç sec that we can get the direct product of such groups with an embcdded rectangular Selnigroup iii tlle slnallest ideal as well. Theorem 3.14. Let T be a compact right topological sew, igroq», let D be a rect- a9u.lar su.bsemigroup of K(T), ad let G be a vm'inal subgrou p of K(T). There is an algebr«ic copy of D x G covtaived "i K(T). Proof. Let L be a lnilfimal left i(leal of D and let R be a lninimal right ideal of D. Since D is reçtangula.r, D is the internal direct product of L and R, lnealfing that each element x of D can be written mliquely as x = xx where x G L and xn G R. Also, RL = R L is a subgroup of D and so, sinçe D consists of idelnpotents, RL = {e} for SOlne e. Then for any x,y D, ,rnyc = e. Note a.lso that (xy) = xc and (.ry)n = y. e lnax" assume th«tt G = etc. Dofine : D x G K(T) by (x, g) = 'Lg,rR. claire that is an injective honlomorphism. Let (, 9), (Y, h) 6 D x G. Then ç((x, g)(y, h)) = xLg.rnyz hyn = .r Lgehyn = (,ry)Lgh(.ry)R = ç(,ry, gh ). Now assume that (x, g) = 9(Y, h). Then g = ege = ,rRxrg.rR.rL = .rnyrhyR.rL = che = h. Also, xLT yLT ¢ 0 and xLT and yLT are minimal right ideals of T; so xLT = yrT. Similarly T.rn = Tyn. Now .r = 'L,rR G .rrT Tan and yLT TyR. So ,ç and y are idempotents in the saine gronp and therefore ,r= y. Corollary 3.15. K(N) cotains an algebraic copy of 1, x F, whe I5, is the 2 « x 2 « rectangldar .semigroltp ad F is the free group on 2 genemtors. Pmof. Bv Corollary 3.13, K(N) contains a copy of the 2 x 2 rectangular Sllli- group, and by [4], each maxinlal group in K(N) contains a copy of the free group on 2 generators. Therefore the result %llows Kom Theorem 3.14. We now present, a strengthening of Corollarv 3.10, producing a longer chain of rectangular components. ecall tha/ the Sosliz zumber S(X) of a topological space X (also known as the cellalarity of X) is the least cardinal 2 such that. X do no/ bave a collection of 7 pairwise disjoint nonempty open subsets. Sec [2, Chapter 12] for considerable information about the Souslin number of t.he space U(n) of uniform ultrafilters on -. ecall. in particular, t.hat the Souslin number of N* = çNN = U(N) is Theorem 3.16. Let be an infinite cardinal and let be an infinite ordinal for c I[ < S(U()). There ezist a set A with lAI = 2 " ,,d , to g if p = q+n for some < w, then 9(g)'9(x) = 9(gx). Also, contains a semigroup D = p< Op of idempotents whe for each p < , Op is a rectagular comportent of D isomorphic to ç, 9[Cp] ç Dp, and the sequence {Dp)p< is decasin 9 in s.ccessor, then D_ ç K(). Proof. Since lAI < S(U(n)), choose a fanfily {Ep)p< of subsets of such that each [Ep[ = n and Ep EqI < ¢ when p ¢ q. For each p < A we define 2808 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK by bp(w) = Yeoepns,pp(,) e (where e0 e = 0) and let be the continuous extension of çp. If v, u, 11 and supp(v) supp(w= 0. then 0p(v + u,) = çp(v) + 0p(w). So by [6. Theoreln 4.21] the restriction of p to N is a hollloHorphisllL Next observe that for x N, p(z) {0}UN. Ifthere exist B z and o < n such that supp(u,) Ep = 0 whenever w ¢ B and rein supp(u,) then ç(x) = 0 because çp is constantly 0 on {u' ¢ B rein supp(u,) Otherwise {0p[{« e B rein supt)(w) OE }1 e - .d < } has the finite intersection property and so is contained in ail ultrafilter . This N and Let T0 = N c?{w II supI)(U,) ç E0}. Notice that T0 is a compact subsemigroup of N. For each p with 0 < p < A let T o= {x Ç N çp(x) e N and for all q with p < q A, çq(w) = 0}. To sec that T v ¢ , let .r be a mfiform ultrafilter on {e a Er. If q ¢ p, then [Eq Er[ < . So çq(.c) = 0. while çp(x) = z N (because çp is the identity on {« a: Ep}). Since q is a homomorl)hisln on N for each q A we has'e that Tp i8 a s/l])Sellligrollp of . Sillce Tp : -1[] p<qA compact. If A is a successor, let T_ = N v v-[N]. Then T_ is clearly compact subsemigroup of N provided T_ ¢ 0. XX show in fact that T_. Let x K(N), let p < A. and let 9 be a uniforln ultrafilter on {e a G Ep}. By [6, Theorem 4.39] pick z G N such that m = z+9+w. Then 0,«) = ç,:) + ç,u) + ,) = ,) + v + ,) ¢ 0. Next observe that for p, q < A, T v+ Tq = Tpvq and if p ¢ q, then T Let T = p< Tp. If T has the relative topology induced bv N, T is a right topological senligroup. Let A = N cg {e " E0}. Then A is exactly the set of uniform ultrafilters on {e "a E0}, and so IA : 2 2. Let C = Ca.. X% shall now construct a smjective homomorphisn f T + C XX first intro- duce some mappings. Let 0 II {e a 0} be a flmction whose restriction to {e a E0} is the identity, whose restriction to {e a Ç E} is a bijection. and whose restriction to II {e a E0 U E} is a bijection. (In particular. 0 is at most three-to-one.) Let 0 1I d {e a E0} be the continuous extension of 0. Let e(0) = a(0) = 0. For w e I1{0}, let e(w) = e where 7 = max supp(w). If supp(w) ç E0, let a(u') = 0. Otherwise let a(,) = e where o = min(supp(w)E0). t - + {0} d{« - < «} ,,a - d {0} u d the continuous ext.ensions of a and e respectively. Notice that a is the identity on {e " O e E0} alld e is the identity on {e " a < }. So ? is the identitv on N d {e a Ç hE0} and 7 is the identity on N cg {e a < -}. X claire that for «. G -¢(x + y) = è(y), (,) (.r+B) = (x) ifxTo, (y} if.rTo. IRECTANGULA1R SEIG1ROUPS IN STONE-('ECH COMPACTIFICATIONS 2809 For ¢v.v E II,, if max supl,(V) > max supp(w), then ((w + v) = e(v) o that è(w+V) : ê(U): if supp(w) ç E0, then 6(u,+v) : 6(v) so that (w+g) : (U). For w, v G II', if max supp(w) < lllill supp(v) and supp(w)E0 # 0, then a(tv + t,) = d(w); so à(w+ ) = d(w). The equations in (*) then follow bv the continuity of Vr . , . .a." = a(g(.,-}). , < p < and .r G 2;, let because & is at most three-to-one.) Thus fiT0] and f[Ts] ç Çs if w 5 P < A. ({ - p < A and let (.e,p) Çp. Ifp < w, let q = p+ 1; otherwise let q = p. Pick g' c( {e e E } such that #(j) = . Pi«k .r Tv. Then ,' + a" + : Tq and /{' +. + ) = ({({{' + + )).(«{' +. + )),p) = = (('), (:).) = (, ,). Whl-«O-,/[d = C. The verification that is a holnomorphisln is routine using the equations (,). Çhoose Ç T 811d (Dp)< gua.ranteed by Theorem .5. Since we have alreadv observed tha lAI = 2- . a.ll conclusions follow ilnlnediatelv except the assertion tha D_ ç K() when A is a successor. To see this recall that K() Ç T_ 1 8o tl18 K) Theorem 8.5(iv), [Ç-I] thus an idem of D and therefore D_ = K(D) Corollary 3.17. Let A be an ordinal for which ]AI = c. There exist a set A with lAI -- 2 c and an injection g : C.4.) --, IHI such that if q < p < A, y E C e, ad .r Ç C v, th« g(a').g(y) = g(a'y), a,,d if p = q+ n for som« n < , tb«n g(y).g(,) = g(y.r). Also, contains a semigroup D = p< D v of idempotetts where for each p < A. Dv is a rectangular «onwonent of D. g[Cp] çDv. a,,d the sequen«e (Dp)v<x is decreasing in the ordermg of components. For each p < A. [DpI = 2 « and f A is a successor, then D_ ç K() ç Pro@ Bv Theorem 2.2, NI and NI are topologically and algebraically isomori)hic. Also S(U(w)) = c +. So this is an ilnmediate conseqllence of Theoreln 3.16. [] It was shown in [5. Corollary 3.4] that there is a _<L-chain (Uo-)o-(tOl of distinct idempotents in ON with the property that for each o- < w, U«+l _< 'u«. \Ve are uow able to establish a considerably stronger statement. (The necessity in the following corolla.ry was also established in [5], but xve includc the short proof for completeness.) Corollary 3.18. Let be an ordial. There is a <i-chain (u«)«< of distinct idempotents in fin with the lnvperty that for each a < A. l/a+1 __ tic if attd onl'!l 2810 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK ,if lAI _< «. /f lAI _< « ard is a ,,.«««so. o,.« ,,, cho.»¢ «h a .,«q««¢ ith Proof. Necessitv. For each « < A, N* + .« properly contains the compact set N* + «+. So one can choose a clopen subsct U« of fin with N* + U«+l ç U« and (N* + u«)U« . The clope subsets of 3N correspond extly to the subsets of N ad so there are exactly « of them. Suciency. Choose A and 9 as guaranteed bv Corollary 3.17 for A. For each p < A, let u v = 9(0.0.p). If A is a successor, then ua-1 9[Ç-] ç DA-I ç ç(). Question 3.19. Is therc a dccreasing 5-chain of idempotents in N* indexed by w+ 17 X close this section by observing thnt it is consistent with ZFÇ that there are kempotents in N that are hot members of any notrivial rectangular subsemigroup of N. Iadeed, by [6, Theorems 12.19 and 12.29 and Lemma 12.44], Marth's Axiom hnplies that there is nn idempotent p fin such that. whenever q N, r ,=l«B(t), and p= q+r, one must havep = q = r. In particular. f p = p+ q +p, then p = q. I can be shown h ZFC that there are idempotents p in that are strongly right maxhnM; .e., the e(luato q + p = p, vth q N, imples that q = p [6. Theorem 9.10]. If p s an dempotet of this kind. p does hot belong to any semigrop h isomorphic o a semgroup of the form 1/ mdess [AI = 1. . CHAINS OF RECTANGULAR SEMIGROUPS AS CO-RETRACTS It was shown in [10] that certain infinite chains of finitc rectangular senfigroups are absolute co-retracts. X prove in this section a shnflar theorem in which the the rectangular semigroups are allowed to be infinite. As a consequence, we obtai additional semigroups whk.h can bc algebraically embedded in . Definition 4.1. Let A = (A}< and B = (B.}< be sequences of sers. Assume that each A, bas a designated elcment a and each B bas a designated element 5. Suppose also that, for each, < w, either A, = {a} or B = {5}. For each p < we define Dp to be the set of pairs of words of the form (a0al -.- av, bvbv-1 .. bo), where ai Ai and bi Bi for each i {0.1 ..... p}. For 0 < A w, we let D.n. = < Dp. We define a semigroup operation on DA.n. as follows: if .r = (aoa...ap, bpbv_...bo) Dp and g = (coc...c,dd_l...do) D, where q p, then xy = (0al-.-p, bp---b+ldd_l-.-d0) and g3 -- (CoC1 " cqaq+l.., ap, bpbp_l .-. bo). We leave the verification that the operation is associative to the reader. Observe that each Dp is a rectangular semigroup. Notice that if A is a set with designated element 0. A0 = {0}. B0 = A. A1 = .4, B1 = {0}, and A, = B, = {0} for n > 1, then D.4.., is isomorphic to CA.,» (Send (0. a) to a and for p > 0 send theelement (0a00---0,00-.-0b) ofDp to (a, b. p -1).) Thus the structure of D.ts., is, in general, considerably more complicated than that of RECTANGULAR SEMIGROUPS IN STONE-CECH COMPACTIFICATIONS 2811 Definition 4.2. Let p < w and let x = (o0o 1 "''(lp, bpbp_l ... bo) E Dp. We define elelllellt.s 1() and 2(,T) in Dp by çl(X) = (01 "..p-l(tp,pp-1 ""0) and 2(X) = (GOal ""ap, bp5p-15p-2"'" 50) and if p > O, we define .re and x in Dp-1 bv xe = (oOal...Op_l, p-lp-2"" o) and x = (o1 ""p-1, bp-lbp-2"'" bo). put u v = (o"" 6p, 5V5V--l''" 0) D v. show that D is somet.hing like an absolute cretract.. Theorem 4.3. Let A = (A)n< ad B = (B)n< be sequences of sets o.s i Defiitio .1, let 0 < A w, artd let D = DA... Let T be a rigt topological semigroup, a«td let f " T D be a sue«tive owmorphism such thot f-l[Dp] is copact for eac p < A. Ten tere is a homomorpism g D T for wich f g is te ide, tity. ff T is compact atd A < w, ten g[D-l] Ç K(T). Proof. mav assume that A0 = {a0} so that D0 is a right zero semigroup. Exactly as in the first paragraph of the proof of Lemma 3.4 we can define g D0 + T such that g is a homomorphism and f g is the i<lentity on D0. So we assume that p > 0 and g has been defincd on q<p Dq. For each « Dp, note that «Dp is a minima.1 right i<leal of Dp ami Dpxis a minimal left idem of Dp. So we mav «hoose a minimal right ideal R(«) of f-l[Dp] and a minimal left idem L(x) of f-l[Dp] such that f[R(«)] = XDp and fiL(x)] = Dp.r. Givcn x e Dp, we have by Lemma 3.3 tha[ g(xe)R(¢(x)) is a minimal right ideal of f-[Dp] and L(¢2(x))g(x) is a minimal left ideal of f-l[Dp]. So we may define g(x) to be the identitv of the group g(xe)R(¢(x))L(¢2('))g(xr). Notice that if T bas a smallest idcal (in particular if T is compact) and A = p + 1, then h-(W) ç f-lirai. S+ ç(f-[V]) ç ç(W) .,,d tm g[V] ç(W). +w f(a(«)) e ,.¢(«); = « a,,d f(a(«)) e ;¢(«)' = «. S+ is an idempotent in 'DpX and thus f(g(x)) = x. Suppose that x e Dp and y e Dq where q ç p. Then ¢(x) = ¢1('Y) and ce = (')«. s+ a(«) e (««)(¢1()) ,d a(«)a() e ('e)(¢l(«))(p) ç g(xe)R(¢(x)) by Lemma 3.3. Thereforc, g(xy) and g(x)g(y) are members of the saine nfinimal right ideal of f- [Dp]. If < , ¢() = ¢() d () = «. s+ («) e z(¢(«))a((:)) a(«)a(u) e L(¢(:))(:)a(u)= (¢:(:))a(:)= (¢(-))a((-)). If = p, ¢() = ¢(y) a,,d («) = y.. S («) e (¢:(y))a() me minimal left ideal of f-[Dp]. By a left-right switch of the al»ove arguments we have that g(yx) and g(y)g(x) are in the saine minimal left idem and the saine minimal right idem of f- [Dp]. First assume that q < p. Pick a e R(¢(x))L(¢2(x)) such that g(x) = g(xe)ag(xç). Thon g(x)g(y)g(x) = g(xe)ag(x.r)g(y)g(xe)ag(x,) = g(xe)ag(x,y.re)og(x,) = («)g(xxe)(xT) = g(xe)ag(x)g(xe)ag(x,) = g(x)g(x) = g(x). So g(x)g(y)g(x)g(y) = g(x)g(y) and g(y)g(x)g(y)g(x) = g(y)g(x) and thus g(x)g(y) = g(xy) and g(y)g(x) = g(yx). Now assume that q = p. Assume also that Bp = {5p}. (The case that Ap = then proceeds by a left-right switch of the following argument.) Then ¢2(x) = '2812 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK _( = . A.o . = .« = ._. Th,,, 9(.)9() L(:(.r))9(x)g(y«) = L(up)g(tp_) and g(y)g(xt) L(up)g(tp_), which is a lninimal lcft ideal of f-l[Dp] by Lelllnla 3.3. ' have alleady verified (ha(g(.r)g(ye) alld g(y)g(xe) are idempotents. So, since they are idempoten/s in /he saine minimal lefl ideal (.)(y«)()(«e) = ()(e). we bave (:.)(.q)g(.) = g(..)(,..)(xt.r) = (.)g(«)g()(..)g(.) = (.)(.'»'t)(.) = g(..,»)g(.«) = g()g(.) = g(). (',onsequently, g(.r)g(y) aml g(y}g(x) are idempotonts. Corollary 4.4. Suppose lhal nis a infi,ite cardiml and lhat eoch «ith«r {0} or 2 2. Thon DA.. tan be embedded in. . Proof. N»r each p < , we give Dp t]le topology defined by regarding Dp as a subspace of (fin 2p+2. We define the topology of D bv ta.king each Dp to be clopen in D. Then D is a topohgical semigroup vith a dense subspace of cardinality The conclusion then follows from Theorems d.3 and 2.5. I EFERENCES 1. J. Berglund, H. Junghenn, and P. Milnes, Analysis on semigroups, V'iley, N.Y., 1989. MR 91b:43001 2. r%V. Çomfort ai,d S. Negrepoiitis, The theory of ultrafilters, Springer-Verlag, Berlin. 1974. MR 53:135 3. L. Gillman and NI. Jerison. Rings of contznuous functions, van Nostrand, Princeton, 1960. MR 22:6994 4. N. Hindman and J. Pym. Free groups and semigroups in/3N, Semigroup Forum 30 (1984), 177-193. MR 86c:22002 5. N. Hiidmai and D. Strauss, Chams of idempotents in pN, Proc. Amer. Math. Soc. 123 (1995), 3881-3888. MR 96b:54037 6. N. Hindman and D. Strauss, Algebra in the Stone-Cech compactification: Theory and appli- cations, de Gruyter, Berlin, 1998. MR 99j:54001 ï. D. McLean, ldempotent semigroups, Amer. Math. Monthlv 61 (1954), 110-113. MR 15:681a 8. J. Pym, Semigroup structure zn Stone-Cech compactifications, J. London Math. Soc. 36 (1987), 421-428. MR 89b:54043 9. W. Ruppert, Rechstopologische Halbgruppen, J. Reine Angew. Math. 261 (1973), 123-133. MR 47:6933 10. Y. Zelenyuk. On subsemigroups of/31 and absolute coretracts, Semigroup Forum 63 (2001), 457-465. MR 2002f:22005 FACULTY OF CYBERNETICS. IYIV TAIAS SHEVCHENKO UNIVERSITY. VOLODYMYRSKA STREET 64. 01033 KYIV, UKRAINE E-mail address: grishko@i, com.ua TRANSACTIONS OF THE AMER1CAN MATHEMA'FICAL SOCIETY Volune 355, Nurnber 7, Pages 2813-2828 S 0002-9947(03)03282-3 Article electronicMly published on March 12. 2003 GALOIS GROUPS OF QUANTUM GROUP ACTIONS AND REGULARITY OF FIXED-POINT ALGEBRAS TAKEHIKO YAMANOUCHI Dedicated to Professor Masamichi Takesaki on the occasion of hzs seventieth birthday ABSTRACT. It is shown that, for a rninimal and integrable action of a locally compact quantum group on a factor, the group of autornorphisrns of the factor leaving the fixed-point algebra pointwise invariant is identified with the intrin- sic group of the dual quantum group. It is proven also that. for such an action, the regularity of the tixed-point algebra is equivalent to the cocommutativity of the cluantun group. l. INTRODUCTION When given an action o of a locallv COnlpact quantum group G on a von Neu- mann algebra A, one may associate to it the subgroup Aut (A/A ) of all autonlor- phisms of A leaving the fixed-point algebra 4 ' invariant pointwise. Let us call this subgroup "the Galois group of a". As [1, Theorem III.3.3] suggests, it would sometimes happen (or be expected) that the Galois group carries an important piece of information on the quantmn group G itself. With this philosophy in mind. we started in [17] to investigate Galois groups of lninimal actions of coinpact Kac algebras on factors by making good use of the Galois correspondence established by Iztuni, Longo and Popa [7]. In [20], we succeeded in describing the Galois grottp of any minimal action of a compact Kac algebra as the so-called intrinsic group of the dual discrete Kac algebra. This extended the result of [1] cited above. As an application, we were able to show that. if the quantum group in question is finite-dimensional, then its cocomlnutativity is equivalent to the regularity of the fixed-point algebra. Our main goal of this paper is to extend these results to a larger class of locallv compact quantum groups, hOt only compact Kac algebras. If we trv to achieve this goal exactlv along the line carried out in [17] and [20], then a Galois correspondence for a (lninimal) action of a more general locally coinpact quantum group would certainlv be needed. At the moment, it seems that the re- sults of Enock in [4] would answer this purpose. Unfortunately, there are however a few lnistakes in his proofs, and. to the best of the author's knowledge, they have hot been restored yet. So we cannot apply Ellock's Galois correspondence to the situation we will consider in this paper. Therefore, we will adopt a new approach here that does llot resort to anv Galois correspondence. Received by the editors June 24. 2002 and, in revised form. November 6, 2002. 2000 Mathematics Subject Classification. Primary 46L65; Secondary 22025, 46L10, 81R50. hey words and phrases. Locally compact quanturn group, action, factor, regularity. @2(103 American Mathematical Society 213 2814 TAtxEHIKO YAMANOUCHI The outline of this paper is the following. In Section 1, we fix the notation used in the whole of our discussion. Basic fa«ts about, locally COlnpaet qllalltUlU groups (iii the sense of Kusterlnans and Va.es) aud their actions on von Neumann algebras are collected. In Sect.ion 2, we will prove that the Galois group of a milfimal, int.egrable action of a locallv compact quantum group G is topologically isomorphic to the intrinsic groul) of the dual G. Section 3 is concerned with regularity of the fixed-point algebl'a of a lninimal, integrable action. We prove that the regularity considerably deterlnines the structure of the qUalltUlll grollp. Naluely it is shown, with some exception, that the fixed-point algebra is regular if and only if the locally colnpact quantum group under «onsideration is cocomnmtative. In Section 4. we lnake a few renmrks on the Izmni-Longo-Popa Galois correspondence. Olm of them concerns an explicit formula fol" the inverse lnap of their Galois correspondence. Finally, we includc an Appeudix fol" SOlUe auxiliary results which are applied to the argument lnade iii Section 3. The author is grateful to Professors Michel Enock and Stefaan Vaes for having ilfformed hiln that there are mistakes in SOllle proofs iii [4]. He is also indebt.ed to Professor Masaki lZUlni for indicating an crror in the earlier draft, of the manuscript. 2. TERMINOLOGY AND NOTATION (àiven a VOll NetlUlallll algebra A and a faithfld llOrlllal semifinite weight ¢ Oll A, we introduce the subsets n, rn and rn of A by * + = me N A+. n = {x A" ¢(x*x) < oe }, me = rionS, rn, The standard (GNS) Hilbert space obt.ailmd frOlll ¢ iS denoted by H. We use the sylnbol AO for the canolliCal embedding of ne# into H 0. The modular objects such as the lnodular operator, the lnodular conjugation, the S-operator, the F- operator, t.he modular automorphism group, etc. associated t.o çb are denoted bv Ve, 34,, S¢, F¢, a ¢, respectively. (Since we follow the notation enlployed in [10], the sylnbol Vwill be used t.o deuote the lnodular operator of a weight.) The set of unitaries iii A is denoted by H(A). For a von Neulnallll subalgebra B of A. define JV'(B) := {u H(A) : uBu* = B} and call it. the normalizer (group) of B in A. V:e let B(H) stand for the algebra of ail bounded operators on a Hilbert space H. 2.1. Locally compact quantuln groups. lï)efinition 2.1. Following [10] (see [9] also), we sa,v t.hat, a quadruple G = (]il, A, 0, b) is a locally compact quantum group (in the von Neulnann algebra sett.ing) or a von Neumann algebraic quantum group if (1) /11 is a von Neumann algebra; (2) A is a unital normal injective .-homomorphism from M into M ® M sat- isfying (A ® id) o A = (id ® A) o A: (3) o is a faithful nornml selnifildte weight on kl such that ri1+ - o((w®zd)(A(:r)))=o(z)w(1) (VwM. +, V:e ), (4) /, is a faithful normal semifinite weight Oll Al such that. /,((id ' w)(A(:r))) = #,(a:)(1) (Vw M. +, Vin më). GALOIS GIqOUPS 2815 Let 118 fixa localh, compact qtlalltllln grotlp Ça; : (]ll, A. ç, b) throughout the rest of this section. We will always think of M as represented on the GNS-Hilbert space H obtailied ffoto . By the left invariance of on H H characterized by II'(G)*(A(x) @ A(y)) = A$(A(g)(,r @ 1)) (x,g n). This unitary is called the Kac-Take.soki operator of G, and is denoted silnply by Il if there is no danger of confllsion. The modular operator and the lnodular conjugation of will be denoted simply by Ç a.nd J. The unitarg atipode, the scalig gronp and the .scalig co.tmd of are respectively denoted bv R, {7t}tER, p() 0). As iii [10], wç aS81llll that @=ço R. According to [10]. there canolficallv exists another locally compact quantmn group â = (kî, , , ), called the locally compact quaztum group dual to a such that {M. H} is a standard lepl'esentation. So we always regard In fact. I is by definition the von Nenmanl algebra gcncrated by {(w @ id)(ll ) w M,}. The lnapping : w M, (w @ id)(lI') I is called the Icft regular repre.settation of G. There is a canolfical identification (= the Fourier trllsforlll) of Hç with H. So we conskler thc lnodula.r operator and the lllOdtllar conjugation of , denoted by Ç and ,Ï, as acting on H. The mfita.ry a.ltipode, the scaling group and the lnodular elelnelg of are denoted respectively by , , . say that G is COlnpact if (1) < . In this case, we agree to take to be a state. X say that G is discrete if G is conlpact. For the defillitions of locally compact qllalltUln groups such as the COllllllllta.llt t, the opposite G °p etc., we refer the readers to [10, SectiOll 4]. It is known that every locally colnpact group F canollically gives rise fo a COlll- nmtative locally conlpact qtlalltlllll group whose tlnderlyillg VOll Netunalln algebra is L(F). XVe denote it by G(F). The underlying von Neulnalm algebra of the dual G(F) is the group von Neunmnn algebra of F gelmrat.ed bv the lefl regular repl'esentation of F. denote by /G(G) the set of all unitaries u I satisfying &(u) = u @ u. The group IG(G) is called the intrinsic group of G. Next defille Ç(G) to be the group ofall automorphisms fl of I such that (fl@id) o& = off. By [2] (see [19] also), /G(G) is topologically isolnorphi« to Ç(G) through the mapping: v IG() fl := Ad v[ Ç(G). Here IG() is endowed with the strong-operator topology, and, for a general (separable) von Neumalm algebra P, we consider on the automorphisln group Aut (P) of P the topology of silnple COllVergellce Oll the predual. It is kllOWll that v is the canonical implelnentation of fit,. ç soEv that N Ç I is a right co-ideal (von Nemnalm subalgebra) of G if is a von Neumann subalgebra of I satisfying &(N) ç N @ ,I. Thanks to [4, Théorèlne 3.3], we know that N Ç I is a right co-ideal of G if and only if Olle has (2.1) N = Ma(MaN')'. A (le) action of G on a von Neunmlm algebra A is a normal injective unital -homolnorphism a from A into kI @ A satis'ing (id ([11). Fix an action a of G on a von Netllllllll algebra A. By [16. Proposition 1.3], the equation %(.):=($ia)((a)) (.+) 2816 TAKEHIKO YAMANOUCHI defines a faithflfi normal operator-vahmd weight T, fl'OIll 4 onto .4 := {a E 4 - c(a) = 1 ® a}, the fixed-point algebra A of c. \Ve call T the operator-valued weight associated to the action o. The ClOssed produ«t of A bv the action is by definition the von Neulnann algebra generated bv c@4) and ]Î c. We denote it. bv G « < A. By [16. Propo- sitioli 2.2]. there exists a unique action a of op on G a < A. called the dual action of a, such that (G«< .4) a = n(A), &(z ( 1) = A°V(z) ® 1 (z e M). For everv faithful normal semifinite weight b on A. by using the operator-valued weight T associated to the duai action d, lhe equation By [16, Theoreln 2.6], therc is a unital ,-isomorphisln O from the double crossed l)roduct (°Pa < (Ge < A) onto B(H)® A. and an action d of G on B(H)® 4 such that g := Ad (El'*2 ® 1) o (er id4) o (id(B,,) ® a). (Ad (Jj) ® 6) ) o = d o 6). where 1" := (J ® J)EII'*z(.Ï .Ï) and : H, ® H --, H v 0¢ H is the flip. \' call à the stabilizatio of a. We say that the a.çtion a is integrable if T is semifinite. The action a, is said to be minimal if AV/(A) ' = C and the linear span of {(id"3ca)(a(a)) : o A. ,,_, G A,} is o--weakly dense iii M. Finally, G is called a Kac algebra if ri = id a.nd er4' = aV. For the general theory of Içac algel)ras, refer to [6]. 3. REALIZATION OF INTRINSIC GROUPS IN AUT (A/A ) Given a von Nelllllallll algebra P alld a von 1Neulnallll subalgebra Q of P, we define Aut (P/Q) t.o be the group of ail automorphisms of P leaving Q invariant pointwise. Let = (M, A, ç, ) be a locally compact quantuln group. Suppose now that we have an action a of G on a von Neulnanl algebra ,4. Throughout this paper. A is alwavs assumed t.o be a non-type In factor (n N). The lnapping considered in the following proposition is essentially observed bv Enock and Schwarlz in [5] as a special case of their constructions of certain lnor- phislns associated to an action of a Kac algebra. The lnapping is still defined even for a general locallv compact quantum group. GALOIS GROUPS 2817 Proposition ;3.1. There e.rists a "uniqtte co,ttinuous hontomorphism front Ç(G) i'rtto Aut (A/A ) su«h that, with O the image of fl ç(G) under this homomorphism, (J d) o., = o If the action o enoys the propert that the linear span of {(idM w)((a)) : a G A, w G .4,} is ¢-'u,eaklp dense i .I. the the above homomorphism is ijective. Poof. net fl ç(G) a,,d ,, e .-. Se X := (d i.«) () e t 4. Then (idM )(X) = (idM ) o (fl idA) o (o) = (d @ id^t = (13 @ id^t = (A®ida ® id4) o (id^l ® ) o o(a) ® idA) o (A ® idA) c(a) ) o (fl ® ia «) o (,,) = (A id From [16. Theorem 2.7], there exists a ,inique element 03(o ) (3.1) (fie id4) o (a) = X = (0fl(o)). It is easv to (']w«k bv using (3.1) Hmt 03 is a hOmOlnorphism rioto A into itself. Moreover, one tan easilv verify that 0d,& = 0d, o 0& and Oid = id. Hen«e 0 is ail automol'phism of .4. That the restriction of 0 d to .4" is the identitv fo]]ows also from (3.1). Thus the lnapping fl G Ç(G) 00 G Aut (A/A") is indeed a homomorphism. Be«ause of (3.1), we find that (fl id4)[(a) is an autolnOllflfiSln. With this in lnind. 0, bas the form 0. = - o (fl Henee fl 0d is continuous. Now suppose that the linea.r si)an of {(idt OE )(o(a)) : G A. G A.} is «-weakly dense in M. If 0 = id. then (3.1) implies that fl is the identitv on {(idt )((a)) :a G A. G A.}. So = id. Consequently. the homomorphism in question is injeetive. Lenmm 3.2. Let fl be an automorphism of 3I such that there is a 0 G Aut (A) such that (fl @id) = 0. Suppose that the lmear san of {(idat ¢)(((a)) : a G A. G A.} is -weakly dense in M. The fl belongs to Ç(G). ad one bas 0 = Proof. Note first that, if 0 is an automorphisln s al)ove, then it automatically belongs to Aut (A/A'). Therefore, it suffices l)v Proposition 3.1 to show tlmt fl belongs to Ç(G). Let a G A and w E A.. Then we have 03 0 id^ ) o /-X( ( id^l /3 satisfies (/3 ® id) o A = A o 3 2818 TAKEHIKO YAMANOUCHI Let be the trivial action of G on C. Namely, t is the mapping frOlll Cinto bI®C defined by,(c) := l@c (c ff C). Then the crossed product G, C is (canonically isomorphic to) Î, and the duM action i is the coproduct o. It is also clem- that the dual weight of tc corresponds to . In this ce. the stabilization ï of t h the form ï(x) = EV*E(1 @ x)EVE for 811y X B(Hw). Therefore, we obt, ain (3.2) 6(x 1) =/(x) , 1 for any x B( H¢). Lemlna3.3. Let z be in B(H¢) such that z@ l G« A. Then z belon9s to M. Pro@ Let z be an element as above. Since (B(H¢) @ A) 6 = G« A. we have 6(z@l) = l@z@l. Onl,eotherhal,d. by (3.2), wehave6(z@l) =[(z)@l. Hence we obtain [(z) = 1 @ z. Since B(H) = M, z must belong to .I. Lemlna 3.4. Suppose that is minimal. Then we bave (A)' B(H) @ A = M' @ C. Pvqf. It is clcm" tha M' @ C is contained in a(A)' B(Hç) @ A. Take anv T a(A)' B(H)@ A. Since T particularly commutes with any element of a,(A ) = C@ A a, it follows from the minimality of a that T belongs to B(Hç) C. So it has the form T = y @ 1 for some y B(Hç). If o ff A and w ff A., then we bave y(id@w)(a(a)) = (idvw)(Ta(a))= (id@w)(a(a)T)= (id@w)(a(a))y. By minimality of a, y nmst be in M'. Since (Ga A) = a(A), it follows from Proposition 3.1 that there exists a contimous homomorphism fl e Ç(o) O Aut (Ge A/a(A)) satisb-ing ( @ id) o = & o 0. Since & enjoys the property mentioned in Proposition 3.1. the homomorphism fl 0d is injective in this case. Lemma 3.5. If the action O Aut (G«< A/a(A)) is Pro@ Let 0 be in Aut (G « cE O(z® 1)(1®b)= is minimal, then the homomorphism fl ff Ç(G °p) topological isomorphism. A/c(A)). If z E M and b A . then O(z® 1)a(b)=O((z® 1)a(b))=O(z®b) O(c(b)(z® 1)) = (1 ®b)O(z® 1). This shows that O(M ® C) __Ç_ (C ® A«) ' = B(H¢) ® (A«) '. From this, we obtain O(M 6; C) ç B(Hc) ® {A C (A«) '} = B(Hc) ® C. Hence, for any z M, there is a torique flo(z) B(H) such that (3.3) O( z ® 1) = flo( z) ® 1. Thanks to Lenlma 3.3, flo(z) belongs to BI. Due to (3.3), it is easy to see tllat ff0 is ail automorphism of M, and that flolo2 = flOl o flo2, flia = id. If a ff A, then (3.4) (flo®id)(&(a(a)))=(Bo®id)(l®a(a))= 1 ®a(o)=d(O(a(a))). GALOIS GROUPS 2819 Fix a faithful nornm.1 semifinite weight w on A, and regard A as represented on H. Let .] be the modulm conjugation of the dual weight & and U the canonical inlplemeltatiol of o on H @ Hw. So U = ,](,1 @ ,Jw). ('hoose the canonical inlplenmntatiol unitary I COlmnUtes with ,L, we also have It follows front Lemma 3.4 that there exists a unitary v I' such that I = v @ 1. Therefore we have flo(Z) = vzv* for any z Ç ;[. By [12, Proposition 1.9], which is valid also for any locally COlnpact quantum gl'OUp, we sec that rt' belollgS to IG(') for some r C with ]r = 1. Since Ioth I" and .ri" are the canonical ilnplementation of 0, we lnust have r= 1. So , is in IG(g'). It follows tiret 0 belongs to çP). Let z M. Then we have a(0(z @ 1)) = (l)o(z) @ 1) = °P(o(z)) @ 1 = (/o id) () l = (o .id) d(» ). From this, together with (3.4), we get (ff0 @ id) o d = d o 0. By Lennna 3.2. we find that 0 = 0 0. Thns we have showll the SUljeclivily. The inverse lnap is also «ontinuous due to (3.3). Theoreln 3.6. If is a minimal, integrable action of a locally compact quantum group = (BI. &. ç, ¢) on a factor A. then there exists a topologicai isomorphsm Ç e Ç(g) 0 e Aut (A/A ) with the property ( @ id) o = a o 0. Pro@ If .4 is infinite, then, by [16, Proposition 6.4], a is a dual action. Hence the assertion follows frOlll Lelnlna 3.5. To deal with a general case, take a (separable) infinite factor L, and put , := L @ A. Also definc d := (« @ id) o (idL @ a), which is an action of on ,4 with A a = L@A infinite. Remark that aut (/2 a) = {id eO. 0 e aut (A/A")}. Let 0 Aut (A/A). By the previous paragraph and the above renmrk, there exists 13 e çg) such that (floEid)o& = 6o(idLO). But this yields 4. I={.EGULARITY OF A IN .4 AS ill the previous section, let cî be a minimal action of a locally colnpact quan- tum group Q = (AI. . ç, ) on a factor A. represent A on a (separable) Hilbert spe K so that {A. K. OE4 } is a standard representation. Let u G (A). Then the restl'iction of Ad u to (A) clearly defines an autolnorphism 0u in Aut ((A)'/A'). The holllOlllorphislll u Ç (A ) Ou Ç Aut ((A)'/A ') obviouslv has H(A ) as its kernel. The bic extension for 4 ç A is denoted by Al. So we lmve A = OE4(A)J4. If A is infinite, then so is A. Thanks to [3, Corollaire 1], we may then choose the above Hilbert space K in such a way that there is a unit vector 0 G K that is cyclic and separating for both A and A . Let OE4 then denote the lnodular conjugation of A associated to 0- Lemma 4.1. Suppose as above that A is infinite. Then the homomorphism u e (A ) 0 Aut ((A)'/A ') defined above is surjective, that is. it bas Aut((A)'/A ') as its image. Pro@ Let 0 be in Aut ((A)'/A'). Since {(A) ', K} is a standard representation, there exists a unitary v on K such that 0 = Adv(A),. Since OA, = id, v belongs to (A')' = A. It is eass" to see that c is in (A). clearly have 0 = 0. 2820 TAKEHIKO YAMANOUCH[ Remark. ff A is finite and G is finite-dilnensional (so that the .bines index fA " A is filfite), then it follows ff'oto [11, Proposition 1.7] that the lllap (A 0 e Aut ((A)'/A ') is smjective. From now on, we asstnne that a is miuimal and itttegrable (A is hot necessarily infinite). Fix a faithful normal semifinite weight « on A. and represent A on H now. Let U be the canonical ilntflementation of a associated to «. Due to [16, Proposition 6.2], this assmnl»tion is equivalent to the one that ( is outer and imegrable. From [16, Corollaly 5.6] and [16, Ploposition 6.2], it follows that the inchtsions C @ A ç a(A) ç G.4 and ,4 ç A ç A are isomorphic. According to [16. Corollm'y 5.6], the isomorphism p" G a A is characterized by (4.1) p(a(o)) = o (a e A), (4.2) p((ç@id)(ll)@l)=(ç@id)(U*) (ç3I.). lncideltally. Equatiou (4.2) can be rewritten as (4.3 p('(ç) 1) = (3Jd3 id)(U) (ç (BI').), where A' stands for the lefl regular representation of G that is given by A() A(3.]ç* J,Ï)*. will make use of this isomorphism in the discussion that follows. ' = 3.4 3, Aut ((W*)'/A ') is isolnorphic to Aut (G Since A = J(A ) J and A ' A/(A)). Thus we obtain a hon,on,orl,hism ff'on, (A ) into Aut (G Bv using the isomorphism p introduced above, it is explicitly given as follows: u (A (*) 0 := p- o Ad (JuJ.)]A o p. As we saw in Lelnlna 4.1 and the remark afler it, this holnomorphism is smjective if A a is infinite or if A is finite and G is finite-dimensional. But it may hot be in general. So our next goal is to identifv its image in detail. For this, first note that. thanks to Proposition 3.5, it is enough to identifv automorphisms fl Ç(G °p such that 03 = 0 for , Moreover. since each 0 has the form 0 = Ad (v @ 1) for a torique v IG(G ) with d = Ad v, it suces to identify unitaries v IG(G') such that Ad (v @ 1) = 0 for some u e W(A ). Proposition 4.2. Let o be a miaimal aud int«grable action of a locall 9 compact quantum gro'up G = (I. A. ç, ,) ou a .factor A. (1) For a, 9 u .ç(A). there ezists a uaique uaitary w(u) IG(G) such that () F (A), ,it w() i t (), , O, = Ad(Jw(u)J 1) = Ad(J N J)a(u)(1N u*)(J N J). We deaote the uitary Jw()J la IG(G') b9 v(u). Pro@ Let u (W). (1) It is straightforward to check that a(u)(1 u*) comlnutes with any elelnent of the forln lça, where a A . So it is contained in BI'Ao(COEAa) = IC. Hence there exists aunitarv w BI such that n(u)(1 )u*) = w@l. Thus a(u) = w@u. By applying A @ id to both sides of this identity, we obtain A(w) = w w. Therefore w belongs to IG(G). So put w() := w. GALOIS GROUPS 2821 (2) Choose/3 E 9(G °v) a.nd t, E IG(G') such that 0u = 03 and = Ad v. Since G' = G °v, we have (A'(0)) = A'(v0). Therefore, by (4.3), the identitv 0 = 0O is equivalent to the lleXt: Ad ( 3& ) ( ,lO3 J id)(U) = (.3vO&ï id)(U) This is flrther equivalent to (,Li &,,L)U(.L) ,LP,L) = (3J )U(,Li« ). By using ff* = (j e &)U(J e ,L), we can rednçe the al,ove identitv to «(u)(1 e u*) = ,Je.] e 1. Consequently, we obtain «(u)= ,lv,l. Definition 4.3. Let /3 be an action of G on a von Nemnalm algebra P. For a. right «o-ideal N of G, the intermediato von Neulnam snbalgebra P(N) of P ç P associated to N (sec [7]) is defined by The fi)llowing lenllna is In'OVen in [4] for the case wh«re G is a. Woronowicz algebra. The clailn and its pl'oof are still valid even for a general locallv compact qua.nt mn group. Lemlna 4.4 (Proposition 3.5, [4]). Let N be a right co-ideal of G. Tken we kove Pro@ Denote by Nt the right-lmnd side of the al)ove clailn. Clearl) we have NI ç N. Let z N. For any 9 N, p B(H). and w 31., we have (p @ w)(( 1)(z) = p(9(id w)((.r))) = p((id @ w)((«))9)= (P @ w')((z(9 1)). This shows that (z) is in N1 N M. Hence (N) ç N N 31. In particular. Ar is also a right co-ideal of G. Therefi)re, := [ defines a.n action of G on If z N. then, by the above result, op(.r) [ @ Nt. Moreover, we have (i« )(;(«))= (i« Z;)(5;('))= (Z ; From [16, Theorem 2.7], it follows that (z) belongs to 7(N) = (A). Thus .r Ç N 1 . Lemma 4.5. Let be an action of G on a von Nemann algebra P. For intenediate von Nemnann subalgebra Q 4 pa ç p. {(id @ ')((z)) " Ç Q. P, }" is a right co-ideal 4G. We denote this rigkt co-ideal bg NA(Q). md call the right co-ideal associated to Q. Pro4 Let 9 e &.(Q)'. For any p, e B(H), w e P. and ze Q. we have 2822 TAKEHIKO YAMANOUCHI From this, we sec that A((zd®a)(/3(x))) is inch,ded in N(Q)® M. Hence N3(Q) is a right co-ideal of G. [] Lemma 4.6. Let 13 be an action of G on, a von Neumann algebra P. |Vith the notation introduc«d above, we bave N = N((GOE P)(N)) for any rigbt co-ideal N of P. If L is the von Neumann subalgebra of OE P generated by (P) and N @ C for some rigbt co-ideal N ofG p, then N(L) = N. Proof. Let N be a right co-ideal of p. It is plain to see that ((G P)(N)) is contailmd in N. Let x N alM p B(Hç),. Since À°P(x) N @ M. we have (.,. )= A»(.) 1 e (g P). Tlms x@ 1 belongs to (GOE P)(N). lfw P, is a state, then (id @ p)(P(x)) = (id @ p@)(P(x) @ 1) = (idpw)(Ç(« 1)) N(( It follows ff'oto Lemma 4.4 that N is «ontained in OE)((G P)(A')). Therefore, we bave prove that N= (( OE P)(N)). Let L be as above. Take any st.are A.. Then, by Lemma 4.4, we hare ,ç = {(id e )(%)) " « e , e ,}" = {(id¢)(h(.) 1)« e »'. e }" = {(d ¢)((, 1))' e X,e a1%}" ç (). In the meantime, L is clearly included in (G OE P)(N). Hence, bv t he result of the previous paragraph, we get N(L) Ç N((OE P)(')) = N. Lelnma 4.7. Let G = (M. A. ç, ¢) be a locallg compact quantum group. Then 1"I 9(G) = C if and only if G is coromtalive. Proof. If G is commutative, then we clearlv bave ]I ç(G) = C. Put N := IG()'. It is easy to see tha.t N is a two-sided co-ideal (von Neumann subalgebra) of .Î (more precisely, of ). Moreover, we have M ç(G) = /I V/N'. From [4, Théorèlne 3.3] (which is still valid for a locally compact quantum group), it follows that N =/Î if/lç(G) = c. Then Ç is cocommutative. [] As explained in Section 1, the mapping v IG(G') Ad vl ï ç,op) is a topological isomorphism. Let /3,. ç(G °p) stand for the inmge of v IG(G') under this isomorphism. Theorem 4.8. Let c be a minimal and integrable action, of a locally compact quan- tum group G = (M,A,ç,,) on, a factor A. ||e set P := .Af(A) '' and define p P to be the basic extension of P C_ A, i.e, P := J, J. Itïth the isomor- phism p" GOEA A, put Q := p-(P). Let w(-)'N'(A ) IG(G) and v(-): .N'(A ) IG(G') be the maps obtained in, Proposition .2. (1) The maps w(.), v(.) are continuous homomorphisros with H(A ) their ker- GALOIS GROUPS 2823 (2) (3) Proof. () We bave Q! = (G OEA) {0:ue'h/(A«)} = (G OEA) {0t-(}: = (G A)(ï{z( :ev()} ). (1) This is straightforward. With the original dcfinition o[ 0, ( E JV'(A")), we clearlv have that (A) {°:ueN(A)} = Pl. So we get the first equality of our assertion. Bv Propo- sition 4.2 (2), we llave 0, = t3,(u), which yields the second equality. The la.st equality follows from the fact that 0o alwavs satisfies (/3 OE id) o & = & o 0 due to Proposition 3.1. (3) This follows fronl Part (2) and Lemma 4.6. Let P Ç Q be an inclusion of von Nemnmm algebras. If the normalizer group (P) of P in Q generates Q, we sv that P is rc9ular iii The next theoronl is a direct generalization of [20, Theorenl 3.6], where we treated only the case where is finite-dimensional. Theorem 4.9. Suppose that a is a minimal and inte9rable action of a locally compact quantum group G = ( M. , ç, ) on a factor A. (1) If A' is regular in A, rb en G is cocom.mutative. (2) If .4 is inflnite, or if A is finite ad G is finite-dimen»ional, then the cocommutativity of G implies the regvlarity of A' in A. Pw@ Retain the notation elnployed iii Theorem 4.8. (1) Suppose that A is regular in A. Then Q = (A). By Theorem 4.8 (3), we get ï {,,() . e (A)}' = {.,:(4} = C. In particular. M {w(u) " (A)} ' = C, because wc bave J3lJ = M in general (sec [101). Sin«e {«(u) u e (A)} '' is a twsided co-ideal of ç, it follows from (2.1) that Hence is cocommutative. (2) Assume that A is infinite, or that A is finite and is finite-dilnensional. Let v IG('). It follows ffoln the proof of Lemma 3.5 that Ad (v OE 1) is in Aut ( oE A/a(A)). Bv assullption, it also follows ri'oto Proposition 4.2 and the dis- cussion preceding it that u e (A ) 0 = Ad (v(u)OE 1) e Aut (ç A/a(A)) is an isomorphism. Hence there is a unique u (A a) such that Ad (v OE 1) = 0 = Ad (v(u) OE 1). By t.he proof of Lemlna 3.5 again, we nmst have v = v(u). Therefore. the map v(-) is surjective, i.e., v((A)) = IG(ç'). So, if is coconlmutative, then v((A)) ' = BI. Fron this, it follows that {z():e(A')} = ;ï M = C. By Theorem 4.8 (3), Na(Q) = C. This means that Q1 = (OEA) a = a(A). Therefore A = P. 2824 TAKEHIKO YAMANOUCHI Next we would like to discuss Theoreln 4.9 (2) iii the case where A is finite and G is infilfite-dilnensional. Let F be a (countable) discrete group and be a minimal co-action of F (i.e.. a niinimal action of G(F)) Oll a type II factor A. For any 7 F, define and call it the eigensubspace of 7- The subspaces {A(7)}er played a vital part in defining the Connes spectrmn F(a) of a in [18]. Proposition 4.10. Let F, ( and A be as above. For any 7 F, tbe eigensubspace A"(7 ) contains a ¢nitary. Pro@ By [18, Theorem 3.17], A(7)*A'(7 ) is a-weakly dense in A «, so that A(?) contains t)lenty of nonzer() elements for any F. Fix an arbitrary 7 F {e}. Put B:= ).. ",bA «, X.]'.4(7 . Bv using *he minimality of a and *he fac* tlmt A' A(?) = {0} (V7 # e), one can easily w'riD tlmt B is a sui)factor of k&(A) = A a&(C). Accordingly. the torique tracial state on A @ a&(C) restricts to that of B. So the projections [(1) a]' [Ô ï] m'e equivalent in B, since they have equal traces there. Hence there exists an isometry }" A(?). Since A is finite. Y is a mfitary. Theorem 4.11. Suppose that c is a minimal, integrable action of a cocommutative locallg compact quantum 9roup G = (5I. X,,) on a II factor ,4. Then .4 is 're9ular in A. Proof. Since A is finite, it follows that G nmst be of COlnpact type. Hence G has the form G : ((F) for a unique (countable) discrete group F. Bv Proposition 4.10. ev- er)" eigensubspace A'(7) contains a unitary I'(7). Cearly V(7) belongs to So it relnains to show that {l'(7)}er and A' together generate A. But. this follows flore the next two facts: (i) A is generated by {A*(7)}er: (ii) A'(-),) has the form A'(7) = A«I;(7) for any 7 F. [] Remark. We remark that, fol" a minimal, integrable action c on an infinite factor A with A ¢* finite, the cocommutativity of G does NOT in general imply the regularity of A . In fact, suppose that A is a factor of type IIIa (0 < ,k < 1). Take a faithful normal state w on A with = idA, where T := -2rr/log A. We regard this inodular action as ail action ct of the one-dimensional torus T on A. It is well known that ct is a minimal (integrable) action. Note that A , the centralizer of w. is a factor of type Iii. Let u be in A/'(A). It is easy to see that u*cz (u) lies in (A")'çA = C. It follows that u belongs to some spectral subspace A(n) := {a A : c,(a) = za (Vz T)} (n Z) of the action c. But, according to [14, Lemma 1.6], every spectral subspace A"(n) except A"(0) = A contains no unit ary. This shows that A/'(A ) is contained in A ". Therefore A is hot regular in A. GALOIS GIROUPS 2825 5. IEMAFIIxS ON ILP's GALOIS COFIFIESPONDENCE Let et be an action of a conlpact Kac algebra G = (11I, A, ç, g) on a factor ,4. In what follows, the action c is aSSUlned t.o ho lninilnal and imegrable, but we de net necessarilv assmne that ,4" is infinite. In [7], a complete Galois correspondelce fol a minimal action of a compact Kac algebra is obt.ained. According te [7, Theoreln 4.4], the map N A(N) gives a one-to-one correspondelwe between the lattice of right co-ideals of G and that of interlnediate snbfac'tors of A C A. Proposition 5.1. The inverse map of ILP's Galois correspotdetce cited above is given by B N,(B). Pro@ Let L, .51 and 6 be as in the proof of Theol-eln 3.6. Then it is easy te check that. for any iltellnediate subfactor B of 4 C_ ,4 and any right co-ideal N of , one has (5.1) .(N) = L ® 4(N), ](L ® B) = (Note that ally intermediate slll)factor C of A' _C has the foliii Ç = L ® B fol a Ulfique/3 as above, thanks te [7, Theorem 3.91 and (the proof of) [15. Lelllllla 2.1].) Therefore, by considering .21 instead of A itself if necessary, we lnay assulnc ff'oin the outset that A is infinite. Then o is dominant by [17, Theoleln 2.'2] or [16. Proposition 6.4]. Se there exists an outer action of Ç' on .4 ¢ such that {.4, o} is conjugate te {Q't< .4 `,/}. Hence we assulne that A = Q't < .4 « g:/ll(| Or = [?. First, by Lelnma 4.6, N = N(A(N)) for any right co-ideal N of . Let B be an intermediate subfactor of A' C A. Choose the unique right co-idcal N such that B = A(Nt) by using [7]. By the result of the preceding paragraph, we obtain A(N(B)) = 4(_N,(A(_N))) = A(]V) = B. This completes the proof. [] The following proposition is regarded as an extension of [211, Theoren 3.5], where we discussed the case of G being finite-dilnelsional. Proposition 5.2. Suppose that a is a minimal action of a compact Kac algebra G = (/11, A,ç,¢) on a factor A with .4 ' inflnite. Then/NÇ(JV'(A) '') = IG(G)". i.e., the right co-ideal erg correspondin9 te the intermediate subfactor JV'(A')" is Ia(C)". Pro@ Let P be the basic ext.elsion ofthe inclusion P := ,V'(A')" C_ 4. With p the isomorphisln introduced just after Lemnm 4.1, set Q := p-1 (p). Bv Lemnm 4.1 and Theorein 4.8, we have Ï ç(ê°) = Na(Q). Frolll this and Corollary 6.3. we obtain Ï ç(-°) =/)(ÏN(P)' ). In other vords, ÏIG(G')' =/)(ÏN(P)' ). Hence, by (2.1), we have N.(P) = AI ç (_î f N.(P)' )' = ]Il {¢(-Ï ç IG(G')' )' = M ç (M ç J IG(G')' J )'= M (M ç IG(G)')' = IG(G)". Thus we are done. Rernark. The above proposition may be used in order te prove Theoreln 4.9 in the case of colnpact Kac algebra actions. 2826 TAIEH IKO YAMANOUCHI 6. APPENDIX Let. a, be a minimal and integrable action of a locally compact quantmn group G = (M, A, , ,) on a factor A. We fix a faithflfl normal semifinite weight w on A and regard A as acting on H. Denote b.v 3 the modular conjugation of the dual weight &. The canonical implementation U of ca, associated to w is then given by U = 3,,(.] ® J,,). The basic extension of A Let. p be the isomorphism p- G,<A A that appeared in Section 3. Identitv (4.2) can be rewritten in the form: (G.1) p(A(¢) ® l) : (¢® id)(U*) Let. A2 1)e the basic extension of A _C At. Since B(H¢)® ,4 is the basic extension of c(A) C G <A. the al»ove isomorphism p can be extended to the isomorphism, still denoted by p. from B(H)A onto A.2. Since B(H)A(CA) ' = B(H)C C, the equation defines a ,-isomorphism il ff'oto B(H«) onto 42 (A') '. Since B(H) 4 (A)' = M' C and G A (C A«) ' = M C, it follows from (6.1) (recall that (6.2) A2 n A' = H(3I'). .4, n(A")'=H(î)={(N.id)(U*).ÇM.}". Since kA(ç))= .lA(ç)*J = A(o R), it follows from (6.1) that (6.3) H((:)) = &H(z)*,L (z e lÎ). Lemma 6.1 (Proposition 4.4. [4]). For any interrnediate subfactor B of ,4 A. N.()' ®C = B(H) ® A ()'. Equiva le, tlg. H (IV (B)') = A._ V B'. Pro@ Since B(H)®Aç(C®.4)'= B(H)®C, wesee that B(H)®Aça(B)'= {T ® 1: T Ç B(H), T ® 1 e c(B)' }. For T e B(H), we have r®lea(B)' <==> (\®¢)((r®l)a(b))=(\®¢)(a(b)(T®l)) ( e B(H,)., ¢ e .4.) \(T(id®¢)(a(b)))= \((id®¢)(a(b))T) (\" e B(H,)., ¢ A.). The last condition is equivalent to T being in N(B)'. [] Lemma 6.2 (Corollaire 4.5, [4]). Let B be an intermediate subfactor of.4 C_ A. (1) We have MCÇ(B)'CC =Q<Aça(B)' , i.e.. n(MA(B)') = AIB t. (2) I4 bave AÇ(B)' C = (M' C) V (Aa(B)'), i.e., A2B' = (A2oA') V (Al I (3) Let B1 :: JwB Jw be the ba$ic extenion of B ç A. and set 1 Then H(R(M AÇ(B)'))= B (A) '. In particular. GALOIS GROUPS 2827 Pro@ (1) This follows from Lemma 6.1 and the identitv Ga<A f-/(C ® Aa) ' = bi®C. (2) This is due to the fact that Nc,(B) = Iii ç? (Iii çl Nc,(B)')'. (3) This follows frolll Part (1) and (6.3). [] Pro@ For w, choose a faithfifl normal state w0 on .4 a and put w := w0 o Ta. By [7, Theorem 3.9], there exists a (uuique) conditional expectation EB from A onto B. Let eB 1)e the Jones projection of B: eBA(a) = A(EB(a)). So BI = (AID {eB})'. Since eB G B1 ç (A(') ', we find that B1 = A V (B ç (A()'). From Lelmna 6.2 (3), it follows that / = c(A) V Ï?(Ï ç/N(B)') ® C. From Lemma 4.6, we find that I EFERENCES [1] H. Araki, R. Haag, D. Kastler and M. Takesaki. Extension of KMS states and chemical potential, Commun. Math. Phys., 53 (1977), 97-134. MR 56:2042 [2] 3. De Cannière, On the intrinsic group of a Kac atgebra, Proc. London Math. Soc., 40 (1980), 1 20. MR 83d:46079 [3] J. Dixmier and O. lvlaréchal, Vecteurs totalisateurs d'une algèbre de von Neumann, Commun. Math. Phys., 22 (1971), 44-50. MR 45:5767 [4] M. Enock, Sous-facteurs internédiaires et groupes quantiques mesurés, J. Operator Theorv, 42 (1999). 305-330. MR 2000i:-16057 [5] M. Enock and J.-1M. Schwartz, Systèmes dynamiques génératisés et correspondances, J. Op- erator Theory, 11 (1984), 273-303. MR 86g:46097 [6] M. Enock and J.-M. Schwartz, Kac Atgebras and Duality of Locatty Compact Groups. Springer-Verlag, Berlin (1992). MR 94e:46001 [7] M. Izumi, R. Longo and S. Popa, A Galois correspondence for compact groups of automor- phisms of von Neumann algebras with a generalization to Kac algebras, J. Funct. Anal.. 155 (1998), 25-63. MR 2000c:46117 [8] H. Kosaki, Extension of Jones' theory on index to arbitrary factors, J. Funct. Anal., 66 (1986), 123-140. MR 87g:46093 [9] J. Kustermans and S. Vaes, Locally compact quantum groups, Ann. Sel. École Norm. Sup., 4 e série, t. 33 (2000), 837-934. MR 2002f:46108 [10] J. Kustermans and S. Vaes, Locally compact quantum groups in the von Neumann atgebra setting, To appear in Math. Stand.. [11] M. Pimsner and S. Popa, Entropy and index for subfactors, Ann. Sel. École Norm. Sup., 4 e série, t. 19 (1986), 57-106. MR 87m:46120 [12] J.-M. Schwartz, Sur la structure des algèbres de Kac, I. J. Funct. Anal., 34 (1979), 370-406. MR 83a:46072a [13] . Str,til,, Modular theory in operator algebras, Abacus Press, Tunbridge W'ells, 1981. MR 85g:46072 [14] M. Takesaki, The structure of a von Neumann algebra with a holnogeneous periodic state, Acta Math., 131 (1973), 79-121. MR 55:11067 [15] T. Teruya and Y. Watatani, Lattices of intermediate subfactors for type Ill factors, Arch. Math.. 68 (1997), 454 463. MR 98d:46071 [16] S. Vaes, The unitary implementation of a locally compact quantum group action, J. Funct. Anal., 180 (2001), 426-480. MR 2002a:46100 [17] T. Ymnanouchi, On dominancy of minimal actions of compact Kac algebras and certain automorphisms in Aut(.A/.A), Math. Stand., 84 (1999), 297-319. MR 2000g:-16101 [18] T. Yamanouchi, The Connes spectrum for actions of compact Kac algebras and factoriality of their crossed products, Hokkaido IVlath. J., 28 (1999), 409-434. MR 2000g:46100 2828 TAKEHIKO YAMANOUCHI [19] T. hamanouchi, Uniqueness of Haar measures for a quasi \¥oronowicz aIgebra, Hokkaido Math. J., 30 (2001), 105-112. Ml 2002e:46075 [20] T. Yamanouchi. Description of the mltomorphism group Aut(,4/,4 c*) for a minimal action of a compact Kac algebra and its application, .]. Funct. Anal., 194 (2002), 1-16. DEPARTMENT OF NIATHEMATICS. FACULTY OF SCIENCE, HOkkAIDO UNIVERSITY. SAPPORO 060- 11810 JAPAN E-mail address: yamanuc@math, sci. hkuda±, ac. jp TRANSACTIONS OF THE AMEIRICAN MATHEMATICAL SOCIETY Volume 355, Number 7. Pages 2829-2855 S 0002-9947(03)03273-2 Article electronically publiqhed on March 14. 2003 COMPOSITION OPERATORS ACTING ON HOLOMORPHIC SOBOLEV SPACES BOO IRIM CH()E, HYUNGWOON KOO. AND WAYNE SMITH ABSTRACT. \Ve study thc action of composition operators Oil Sobolev spaces of analytic flmctions having fractional derivatives in some weighted Bergman space or Hardy space on the unit disk. Criteria for when such operators are bounded or compact are given. In particular, we find the precise range of or- ders of fractional derivatives for which all composition operators are bounded on such spaces. Sharp results about boundedness and compactness of a com- position operator are also given when the inducing inap is polygonal. 1. INTF¢ODUCTION AND STATEMENT OF IESI'LTS Let D be the unit disk in thc complcx plane. shall write H(D) for the «lass of ail holomorphic functions on D. Let .s k 0 be a l'eal muni»er. Following [BB], we defilm the flactiolml del'ivative tbr J'G H(D) of ordcr bv f(z) = (1 + n)az , z D where 0 z is the Tavlor series of f. In this pal)er, we are going to investigate composition operators a«ting on holo- moI'phic Sobolev spaces defined in terlns of fractional derivatives. To introduce those hololnorphic Sobolev spa«es, let us first recall SOlne wcll-klK»Wn flmction spaces. For 0 < p < and a > -1 the weighted Bergman space 4 v is the space of all f H(D) for which = f I/(z)lp( -Izl) dA(z) < ,If,, where dA is area measure on D. Also, the Hardy space H p is the spa«e of all g H(D) for which ]o " dO O<r<l We will oit.Pli use the following notation to allow unified statements: AP_I = H p. TMs notation is justified bv the weak-star convergence of (o. + 1 )(1- [z[) a dA(z)/zr to dO/2zr as a -- -1. tt.eceived by the editors April 4, 2002 and, in revised form, August 6. 2002. 2000 Mathematics Subject Classification. Primary 47B33: Secondary 30D55, 46E15. Key words and phrases. Composition operator, fractional derivative, Bergman space. The second author's research was partially supported by KF/F2001-041-D00012. ()2003 American iNIathematical Societ 2830 B. CHOE, H. IxOO, AND W T. S/vIITH Now, for p > 0, s OE 0 and et >_ -1, the hololnorphic Sobolev spa.ce A, s is defined to be the space of all f H(D) for which *f APo. will often write H = A define the norm of f .4 by [[fIIAL = Of course, we are abusing the terre "norm", since I1 IIAe, does llot satisçv the t.rialagle inequality for 0 < p < 1. but in this ce (f, 9) IIf - 9llA. defines a translation-invarian metric on .4 which turns Ag into a complete topological vector spae. A fraction H(D) that satisfies ç(D) C D induces the composition operator C, defined on H(D) by Throughout fifis paper the symbol ç will always represent a holomorphic self-lnap of D. In this paper we study the action of composition operators on holomorphic Sobolev spaces. This setting allows a unified trea.tment of composition operators _ on Hm'dy spa.ces (H p = .4 p 1.0), weighted Bergman spaces (.4 = P -1), and Dirichlet-type spaces (A,), where extensive resem'ch has already been done. The book [CM] is a good introduction to this work. The main results in this paper may be viewed as smmnarizing well-known boundedness and compactness results for composition operators on these spaces, and then extending them to the Sobolev setting. It is a well-known consequence of Littlewood's Subordination Principle that everv composition operator is bounded on A for every p > 0 and -1: see [MS]. It is natural to ask how this extends to the spaces A. when s > 0. For p > 0, j > -1, sj 0 (j = 1.2) with a - 2 = p(s - s2), we have the following equivalence (see Theorem g.12 in [BB]): (1.1) P A p AI -Sl 2,82 " That is, these spaces are isomorphic and have equivalent norms. In particular, when s < «+ we hae A s Aa_sp. Thus it follows that eve- composition operator is bounded on A s when s < +t. The general situation is described in the following , p theorem. Just the statement of this and out other main results are given in this section. The proofs will corne later. Theoreln 1.1. Let p > O. s O and -1. + ten ever composition operator s bonded on A, s. (a) If .s < , (b) Ifs- + p (i) p 2 or = --I, then eve composition operator s bounded on A, s. (il) p < 2 and > -1. ten some composition operators are hot bounded o (c) If s > + then some composition operators are hot bouded on A The case = -I in part (b) corresponds to s = 0, and previously mention everv, composition operator is bounded on H = 4P-,0- The case o = -1 in part (c) shows that this does hot extend to H for a range of positive s, for the Bergnmn-Sobolev spaces. The boulldS on s in Theorem 1.1 can be extended when the inducing map of the composition operator is univalent or, more generally, of bounded valence. COMPOSITION OPERATORS ON SOBOLEV SPACES 2831 The upper bound given in §6. We do sharp, but another be extended to the The equivalence for oel > -1 with o1 inclusion relations: (1.2) (1.3) Theorem 1.2. Let p > O, s > 0 a,d a > -1. Assume that q is of bonded valence. (a) If p>2 and s < a+ then C is bounded on A . a+l 1 then C is bouttded on A,. (b) If p<2 ands< +, s < a+ inpart. (a) forp > 9 issharp: auexamplewillbe a+l hot know whethcr the ui)per bouud s < + in part (b) is examl)le will bc given that shows that the upper bound cmmot 1)omd «+ from part (a). p (1.1) does not extend to the limiting case ee = -1. However, + 1 = p(st - see), we have the folh)wing Littlcwood-Paley-type p < 2 === 4 p C AP_I,se, -- - (3i , I ,P p> 2 === . -1,s2 C __ - Ol,S I hlclusion relations for different values of p are also kllown. aj > -1, si > 0 (j = 1,2) with +2 +ee = st - see, we have P P2 For 0 < Pi < P2, Let. p > 0, c >_ -1. and s >_ 0. Note that ,are have .4, C A (+2)/(+2-p) for ps < c + 2. as a special case of the above inclusion (s2 = 0, ni = (2)- hl case ps > c + 2, inclusion relations with other types of fimction spaces are known a.s follows: (1.5) 0 < ps - (( + 2) < p == -4Pa,s C As-(a+2)/p, (1.6) ps=c+2 4 p cVMOA. Here, Ae denotes thc holomorphic Lipschitz space of order e, 0 < e < 1, and VMOA denotes the space of holomorphic functions of vanishing meml oscillation. The definitions and more information on these spaces can be round in [CM] for A and [G] for VMOA. For details of ail the inclusions mentioned above, see Theorenl 5.12, Theorem 5.13, and Theorenl 5.14 in [BB]. The boundedness (conlpactness) of a composition operator on a smaller space often implies the boundedness (compactness) of the operator on larger spaces. This general philosophy and the inclusion relations nlentioned above lead to natural con- jectures. The nlethods developed below in §2 to addrcss tllese conjectures require some restriction on the parmneters. In particular, the case (1.2) is left open since out nethods do not apply when the target space is a Hardy-Sobolev spa.ce. Theorem 1.3. Let X C I" be any of the inclusion relation in (1.3) - (1.5), and assume for iclusion (1.3) that see < 1. a, for mchtsio,(1.4) that c2 > -1 and s2< 1+(1+c2)/P2. (a) If C, X --, X is bounded, the Cv }" --, Y is bounded. (b) If C " X X is compact, then C " Y - Y is compact. Inclusio,1 (1.6) was left out of the preceding tlleo,'eln, but we have the following partial result in that case. Theorem 1.4. Let p > O, o >_ -1. s > 0 and assume ps = a + 2. (a) If C, " A, ---, AP, is bounded, then C, VMOA -- VMOA is bounded. 2832 13. CHOE, H. KOO, AND "V. SMITH VIIOA is compact. \Ve also mention the elementary inclusion relations that. for ail p > 0, s >_ 0. c _> -1, and e _> 0, ,tP P In §3 we will give a result analogous to Theolem 1.3 for these inclusions, but with SOllle restrictions on the parameters. As a first application of Theoreln 1.3. notice that it can be used to prove the case c > -1, p > 2. and s = +__! in Theorem 1.1. Then H p CAPa s bv (1.3), and so everv COlnposition operator is bomded on AP, bv Theorem 1.3. hl the other direction, once criteria for Co to be bOulded or compact on the larger spaces are known. Theoreln 1.3 can be used to l)rovide necessary conditions fol bouldedlleSS or COlnpactness of Co on the slnaller spaces. For example, bv taking Ae as the larger space, we bave the flllowing consequence, which has |)een kllown for p _> 2 (Theorem 4.13 in [CI[]), while it bas been known to be false for p = 1 (p. 193 in [Chi]). So. the gap l < p < 2 is now filled in. A more general version is proved as Theorem 3.3 below. Theorem 1.5. Let p > 1 aTtd suppos« C " II' -- Hï is bo«vded. Then the angular derivative of ç exists at all points ç i)D whcre the radial limit (Ç) of ç exists A basic problem in the studv of conlposition operators is to relate function- theoretic properties of ç to operator-theoretic properties of the restriction of C'o to 4 p bv various spaces, as in Theoreln 1.5. When ps < a + 1. we have .4Pa s " a-sp , (1.1), and criteria for (7 4 pi 4 p= to be bounded or COlnpact are klloWll. The O1 2 characterization is that a generalized Nevanlinna colnting function for , satisfies a growth condition if p2 >_ pi, or an integrability condition if p2 < p: sec [Sml] and [SY]. The results in [Sml] and [SY] do hOt apply when ps >_ a + 1 in either the domain or the target space. |11 that case, criteria in the form of Carleson lneasure conditions for a measure defined using a lnodified comt.ing fimct Joli call be obt ained as in Theorem 2.6 below, with sonle restrictions on the paralneters ctj. pj, and si. This Carleson-type criteria in Theoreln 2.6 will be used to prove Theorelns 1.2 and 1.3. \Ve also lnention that for the special case p = 2 other teclmiques m'e available. since the llOrlll of a ftlllctioll in A, lnaV be given in terlns of its power series coeflïcients. These spa.ces are examples of what are called weighted Hardv spaces in [CM], which is a good reference for colnpositiol operators acting on these spaces. Characterizing when a colnposition opera.tor is bounded on HsP, s > 0, seelns much harder. The diflïicultv is that (1.1) does not provide ail isonlorphisnl with a space of functions defined with full derivatives, and the llethods used to prove Theorem 2.6 do hot apply. We have froln Theoreln 1.1 that. for anv p > 0 and s > 0. there exists a functiol ç such that C is hot bounded on Hs p. A positive result is that C is COlnpact on certain Hs p whelever ç is of bounded valence and ç(D) is COlltained in a polygonal region COlltained in D. This is the special case pi = P2 of the following result. Fol" a polygon P inscribed in the trait circle, let O(P) denote 1/rr times the measure of the largest vertex angle of P. COMPOSITION OPERATORS ON SOBOLEV SPACES 2833 Theoreln 1.6. Let 1»2 _> p, > 0 and as.saine 0 <_ s < min{&p2, }'1 Let ç be a h, olomorphic flo.**ctio' of bo'uded vo, lence tat:ig D ito a polygon P iscribed i th u, it circle. IfO(P) < p(1--sp2) thf @ " Hf Hy 2 is çom]ioct. p2 (1- sp ) " When s = 0 and p = P2, this has long been known: see [ST]. When s = 0 and p p, this result is basicallv contained in [Sml]. These results (when s = 0) do hot require the hypothesis of bounded valence. will prove a more general result in Theorem 5.5. In the next section we develop the change of variable methods that we use to study composition operators, which we then use to give Carleson measure-type criteria for these operators to bc bounded or COmlmct. These criteria are then used in 3 to prove Theorem 1.3. Next. in 4. the proofs of Theorem 1.1 and Theorem 1.2 e given. Simple geometric criteria are then developed in 5 for loundedness and compactness of a composition operator between holomorphic Sobolev spaces when the inducing map is polygonal. The pal»er concludes, in 6. with several examples which denlonstrate that out theorems are sharp. 2. ]A('KGttOUND: CARLESON-TSt'E CRITERIA Oto" al)proach t.o studying COlnposition operators on the spaces A, involves a change of variable ri'oto z to w = ç(z). The equivalen«e (1.1) allows us to assume that s is an integer, and then standard non-retiraient change of variable methods can be applied. This gets quite compli«ated when s is an integer greater than 1. Thus, for simplicity and claritv of presentation, we confine our attention to the case ,s = 1. Thisenables us to cover parmnetersp. and s with +(1-s)p > -1 by using the equivalence A, A2+(l_)p.1 ri'oto (1.1). The change of variable method for s = 1 is summarized as follows. For a hololnorphic lnap D D and w G D. define the modified counting function Ç,(ç, w)corresponding to the measure (1- [z[)dA(z) by oEç,(ç, w) = Iç'(z)lP-(1 -Izl) where the sure is over the set {z ç(z) = w}. As usual, the zeros ofç- w are repeated according to their multiplicity. The change of variable formula we need ses the lllestlre Then. bv the area formula (see Theorem 2.32 in [CM]), we have the following «halage of variable fornmla. Proposition 2.1. Let p > 0 atd c > -1. Then, we bave [(f ° ç)'(z)[P( 1 --Izl2)dA(z) = / [f'(w)[Pd#, (w) . (D) for functions f H(D). Note that Proposition 2.1 cannot be directly applied to the case s = 1, because Ttf(z) = f(z)+zf'(z) by our definition. This difficultv is overcome bv the following proposition. 'e will off.en write X < }" if X _< C" for some positive constant C dependent only on allowed parmneters, and X }" if X < }- < X. 2834 B. CHOE, H. KOO, AND W. SMITH Proposition 2.2. Let p > O. c >_ -I and a E D. ri. we bave for f e H(D). Then, for every positive integer n--1 Pro@ We prove the proposition for a = 0. The proof for general a is sinfilar. The eq**ivalece IIIIA. 2=0 Ilfl)llA is proved in Thorem 5.3 of [BB]. Thus, llfl A > n--1 --0 If()(°)l ÷ IIf()llAa is clear by subharmonicity. Now, we prove the other direction of the inequalities. Since H(D) is dense in ail holomorphic Sobolev spaces bv Lemma 5.2 of [BB], it is suflîcient to show that (2.1) IIfllA < If(0)l ÷ IIf'llA, f H(D). First, assume either ve > -1 or 0 < p _< 1. Let. f H(D). For each/3 > -1, we bave bv Theorem 1.9 of [BD], 1 /D Tf(w)Gl(z)(1 -Iwl=)aA(u') f ( ) : - . where 1{1 Ge(Z)=z (1-z)+ 1 . Therefore, choosing ./3 > -1 sufficiently large, we have by Lenmm 4.1 of [BB] (ve > -1 or 0 < p _< 1 is used here). (2.2) IIfllA < IIfllA+ IIfllAX+ ÷ IIf'll o+ A p . It is easy to see tlmt, given g > 0, there exist a constant C > 0 and a COlnpact subset Iç= {z Dlz ] _< r< 1} ofD su«h that [If[l£+ < ellfllA ÷ C sup If(z)l. Taking > 0 sufiîciently small, we have by (2.2), IlfllA¢ £ IIf'llA+ ÷ sup If()l < If(0)l + IIf'llA + sup If(z) -- f(0)l < If(0)l ÷ IIf'llA¢ ÷ sup If'(z)l. Since supe/,- If'(z)l £ IIf'llAg by the subharmonicity of If'l p, we obtain (2.1) as desired. Now, consider the case = -1 and p > 1. Note that [f(e °) - f(0)l < If'(te°)ldt. Therefore, by Minkowski's inequality, we bave [If- f(0)[[H < [f'(te°)[PdO dt <_ IIf'llH». which implies (2.1). The proof is complete. [] COMPOSITION OPERATORS ON SOBOLEV SPACES 2835 Having seen Proposition 2.1 and Proposition 2.2, it is now clear that the behavior of C, when the target space is A p depends on that of the measure ttp, For botm(ledness and compactness of C, the criteria for It, turn out to be Carleson- type conditions in certain cases. To pi'ove it, we ueed a couple of lemnms. Lemma 2.3. A bounded subset of any of the spoces APo, s. a >_ - 1, is a normal fam ily. where p > O. s >_ O. and Pro@ First assume o > --1. Using (1.1), a bounded set X in AP.s is also bounded in some A p where is a nonnegative integer. Recall that there is a constant such that I.q(,*')l for ail g A (see, fol" example, Tlmoreln 7.2.5 in [R1]). By Prol)osition 2.2, this shows that the functioIs in X are mfiforlnly bomMed on COlnpact snt)sets of D. Hence X is a normal family. The proof of the resull fi)l'n = --1 is sinfilar, since .4ç,, æ c .4ûy æ by (1.4). The l,roof is COlnplete. In the next lelmna we will need the estimale that if a > -1 and/3 > 0, then fD (1--11)" dA() 1 (2.3) Il - zl ++ A reference is Theorem 1.7 of [HKZ]. Lemma 2.4. Let p > O. c > -1 and s > O. Let N > +- - s. Put g(z) = (1 - zO) -v for a, z D. Then. we have IlgallAs,. (1 -I1)--+, D where the constants in this estimate depend on N, s. a. and p, but are mdependent ofa. Pro@ First, assume c > -1. Let k be the smallest integer satisfying k _> s. Then, we have AP, APa+(k_s)p.k by (1.1). Thus, by Proposition 2.2, we bave - { fD (l _ D[)'+(k-)PA(z) } /P [ça AP "" 1 + Z cN'j[a[J + CN'k]a[k 11 - zg[P(N+k) where c,j = N(N + 1)...(N + j- 1). Thus, by (2.3), we have ]g AP where C = C(N,s,c.p). The desired estinmte follows. Next, assume c = -1. API A p 2p Note that o,a/p+s C _, C Ao. by (1.4) and thus Ilgoll A,:,..," £ IlgoIIA,.., £ Ilgoll,«/=,:,.:,/,+., O11 the other hand, we have II.qll.g.,,., ( -I,1) --÷ by what we have just proved for the case ct > -1. This completes the proof. 2836 B. CHOE, H. KOO, AND W. SMITH For anv arc I C BD define the Carleson square over I tobe SI = {re ie 1 -II] < r< 1. e e I}, where [1[ is 1/(2rr) rimes the Euclidean length of I. Also, let ô denote the complex differential operator, i.e., ôf = f' for f H(D). The next lemma asserts that certain operators are compact. review the definition, since when p < 1 the spaces involved are hot Banach spaces. Suppose X and }" are complete topological vectors spaces whose topologies are induced bv metrics. A contilmous linear operator T " X }" is said to be compact if the image of everv bounded set in X is relatively colnpact in } Due to the metric topology of }" T will be compact if and onlv if the image of every bounded sequence in X bas a subsequence that converges in }'. Also, linearity of T allows us to only consider sequences in the mfit ball of X. In the following lemma, part (a) is well known: see Theorems 2.2 and 3.1 in IL1]. Part (b) is certainly known to experts. For example, the case k = 0, p = q, and a > -1 ocçurs as Theorem 4.3 in [MS]. A proof is included here since we do hot know a reference. In out application, we will take k 1. Lemma 2.5. Assume that one of the followin9 three conditions holds: (i) a>-l.O<pGq; (ii) a=-l,p=qk2: (iii)«=-l.0<p<q. Let k be a nonne9ative ite9er and p be a positive finite Borel measure on D. (a) ô " " A Lq(d) is bounded if and only if O (]IIq+q(«+)/v) . I C 0D. I+(SI) (b) 0 . A Lq(dp) i+ complet « nd ol if c+.> C+> = + O. Moover. the norm q the nap in (a) satisfies the inequality 11011 + CIlll, IIl tSe supremum of the quanlity <S0/llq+q< +2v o,,e I c OD. Proof. provide a proof of {b). ç first prove the suciellCV. So, assunle and let {f.. be a bounded sequcnce in A, say of norm at lnost 1/2. We must show that {f} contains a subsequence whose k-th derivatives converge in Lq(d}. ecall that we have observed that a bounded set in A is a normal falnily, and so bv subtracting the limit function and re-indexing an appropriate subsequence, We may assaille that 11511A¢ 1 and that {f} and hence {f t converges to 0 uniformly on compact subsets of D. We need to show that {f[)} converges to 0 in Lq(dp). Let e > 0 and write liN& IIL.(d.) = Ifa)[qdp + Ifa)[qdp, 0 < r < 1. The first tenn is easilv handled. For any fixed r (0.1), the uniform convergence of {f)} to 0 on rD allows us to find N(r) such that < D Turning to the second terre, by hypothesis we can choose r = r (0, 1) so that thc mesure dr(w) = XDD(W)dp(w) satisfics v(SI) zI] , whenever [I] 1- r. COMPOSITION OPERATORS ON SOBOLEV SPACES 2837 where fl =/,'q + q(2 + o)/p. For II1 > 1 - r, we subdivide I into m arcs of length at lnost l-r, where 'm 5 [I[/(1-r)+ 1 2Il[/(i-r), and observe that SI(DrD) is contained in the Carleson squares associated with the smaller arcs. Thus. the previous estimate shows that oe(S/) e(1 - r) 2ellI in this cse as well. Note that we used 2 1, which is a consequence of the hypotheses, for the last inequality. Thus, xve sec rioin (a) that there is a constant Ç such that Slll) Combilmd with the previous estimate, this shows that IIf} ) I](d,) 0 as required. Now, wc prove the necessity. Suppose (2.4) is false. Tlmn there exist a constant C2 > 0 and a sequcnce of arcs I, C 0D such that I, I {1 and (2.5) (SI,,) clI,l +(+/. Let = lIl and ( Ç OD be the center of I, fi»r each . Fix a large integer N > (+ 2)/p. Let g,(z) = (1 - (1 - )zÇ,) -N and put f, = gllgA. - Note that [10,,llg a;Np++z by Lemma 2.4. Tlms, {f} converges uniformlv to 0 on compact subsets of D. Now, using the COlnpactness of 0 k A Lq(dp), pick a subsequence of {f,, } whose k-th derivatives converge to 0 in Lq(dl t) and use the saine notation for that subsequence. Note that l1 - (1 - a)zÇ] for z e SI,, and n large. Thus, by (2.5), we bave (a) for ail large z. This is a contradiction, because IIf, I1) 0. The proof is complete. Now, a change of variables md standard m'gumelts give us the following Car- leson measure characterizations of 1)oundedness and compactness. As discussed in the first paragraph of ihis section, we restrict out considera(ion of the orders of differentia¢ion to certain rmlges: mmlvsis of the general case seems too complicated for this paper. also lnention again that when 8p ( ç 1 or p 2, other methods are available and much more is known: see the discussion following Theorem 1.5 in the Introduction. Theorem 2.6. Assume ttat one of the Jbllowing three conditions holds: (i) O' 1 > -1.0 < Pl _ P2: (il) a = -1,pl = Pz >_ 2; (iii) cri = -1.0 < Pi < Also. as.sume cz >-1 ad o 1 ÷ 2 1 a2 + 1 (2.6) 0_<s < 1+ , 0_<s <l+-- Pi P p 2838 13. CHOE, H. KOO. AND W. SMITH A v is bounded if and on lg if (a) c A2,.,--, , (_.,) p,+(l_)p2(SI) : 0 [I[ (2+)p2/p1+(1-s1)p2 I C (b) C A p A p is compact if and only if 181 282 P p2,a2+(1-s2)P2 " Pro@ Here, for brevity, we prove the suciency for boundedness and the necessitv for compactness. The other ilnplications can 1)e seen 1)y ey lnodifications. Also, let p for silnl)licity. : Pp2,a2+(1-s2)p2 First. we provc the siCiCllCy for bomdedness. One may easily modify the proof for conq)actness. So, suppose that p satisfies (2.7). Note that 1)y the first part of (2.6). Thus, by Lelnma 2.5 (a) (k : 0), we have for fimctions g holomorphic on D. Also. note that a2 + (1 - s)p2 > -1 1)y th(' second part of (2.6). It follows flore (I.I), Proposition 2.2. Proposition 2.1 and (2.9) that P2 5 [f(ç(0))l + IIf'll_+(+.l»«,)_. . Now, bv Proposition 2.2 and (1.1) again, we see that the sure in the last line above is equivalent to " P2-- +( al)P2/Pl Next, it is clear that [R s' f(0)[ IIfl[.4,Sl- Also, it is easy to verify using Lemma 2.5 (a) (k = 1) that < I I/IIA«EE = II/IIA:,I II0 '/IIAX_+,+I»» ' Putting these estinlates together, we conclude the boundedness of C " 4 pi - 1,81 2,2 " A p is compact. Suppose that (2.8) does not hold. Then there exist a constant C > 0 and a seqllenCe of arcs In C OD Sllc that IIl 0 and (S[n) OE Cin[ (2+eI)p2/pI+(1-sI)p2 Let = ILI and Ç 0D be the center of I for each n. Fixa large integer X > (a+2)/p s. Lette(z) (1 (1 )zç) - and put f ..... IiIIAx.I" Note [[ff[IA,l -81+(2+1)/1 by LIIII OE,. TlllI, {A } COllVFge unifoFllll}" tO 0 on colnpt subsets of D. Therefore, using the colnpactness of 4 p2 and APa,s=, we mav. pick a subsequence of {f o ç} that converges to 0 in . . COMPOSITION OPERATORS ON SOBOLEV SPACES 2839 use the saine notation for that subsequence. Now, first using Proposition 2.1 and then proceeding as in the proof of Lenlnla 2.5, we have IIf ° llAo,s IIf I[A= 2+(l--s2)P2,1 5 -(+«)/'. Il-(1 >C for all large n. This is a contradiction, because IIf o IIA=. 0. The proof is complete. 3. FROM SMALL SPACES TO LARGER SPACES %'e now turn t.o the proof of Theorem 1.3. For convenien«e we divide the theorem into more easily managed pieces, considering ea«h implication separately as well boundedness and conlpactness. Theorem 3.1. Let pi, si and j (j = 1,2) be as in tbe bypotbeses of Theorem 2.6. In addition, assume tbat - = si - s2. Pl P2 (a) C A m v (compact. resp.) if and onl'y if C is , A., is bounded bounded (compact, resp.) on A 2,8 2 (b) If C9 is bounded (compact, resp.) on Proof. Note that (2 + a)p2/pt + (1 - 81)P2 = 2 + 2 + (l -- 82)P2- Thus, (a) follows v= by (1.4). Thus, (b) follows from from Theorem 2.6. Also, noie that 4 ma,s C Aa=,s = (a). It is straightforward to check that when a = -1 and pi = P2 2. the hypotheses (2.6) in Theorem 2.6 are equivalent to s2 < 1 in (1.3). Thus, Theorem 1.3 with inclusion (1.3) is an immediate consequcnce of Theorem 3.1. Similarly, when -1 and 0 < p < p2, the hypotheses (2.6) in Theorem 2.6 are equivalent to a2 > -1 and s2 < 1 + (1 + a2)/p2 in (1.4), and so Theorem 1.3 with inclusion (1.4) follows. The proof of the next theorem uses some properties of the pseudo-hyperbolic distance p on D. Recall that the pseudo-hyperbolic distance between points a and b in D is given by a -- b p(,b)= ï-2 I We use D(a, r) to denote the pseudo-hyperbolic disk of radius r and center o. Recall also the well-known and useful identity - b 12 (1 -112)(1 -Ibl 2) a,bD. In particular, it is a consequence of this that (3.1) I 1 - bl 1 -I12 1 - I12 whenever b D(a, 1/2). The next result covers the inclusion (1.5) in Theorem 1.3. and so completes its proof. 2840 B. CHOE. H. KOO, AND W. SMITH Theorem 3.2. Let p > O, o > -1 and a+_A2 < s < 1+ a+e If ('o is bounded -- p P (compact, resp.) on Pro@ first prove the assertion on boundcdness with the additional assumpt ions that o > -1, p > 1 and «+ < s < 1+ «+ Note that a+(1-s)p> -1 and P P a p bv (1.1). Choose a D such that ]ç(a)[ > 1/2, and therefore A, m . a+(1--s)p.1 - -- consider the test function foc(z) = log(1 -ç(a)z). Then, by Proposition 2.2. we have / Iç(a)l Zl2)a+(1 [IL çll:+,l_.,.., E I - ç(a)ç(z) Iç'(z)l( -I -'}PdA(z) - For z D(a, 1/2), we have 1 -]z 2 1 -la z, by (3.D. Also, the Schwarz-Pick Lemma tells us that ç(z) 6 D(ç(a), 1/2), and so 1 -ç(a)ç(z)[ 1 -Iç(a) from (3.1). Using these estimates in the lt terre in the display above shows that (3.2) > For the last inequality we used that D(a, 1/2) contains a Euclidean disk with center a and radius comparable to 1 - ]a 2, and that ç'l p is subhannonic. Meanwhile, since f(0) = 0, we have (3.3) where the lt. equivalence holds by (2.3), because sp > a + 2. Putting these estimates together with the assumption that C " A, A, is bounded, we get aDSUp 'p'(a)[ { (1-- 'ç(a)[) } -l+'-(a+)/v(l - This is equivalent to the bomdedness of C on A_(«+)/v; see [Mai or Theorem 4.9 in [CM]. Now, consider the general case o > -1 and a+ < s < 1 + a+. Choose q > p p P so large that q > 1 and s < 1 + a+ . Put 13 - (a+2)q 2. Then. > a > -1 P q P and +2 -- +2 Now, bv (1.4) and (1.5), we have q P 4 Also, note that + < s < 1+ + Now, suppose that C A,» + A is q q bounded. Then C " A, A, is bounded by Theorem 3.1 and thus so is C " As-(a+2)/p As-(a+2)/p by the result for the special case we proved first. This proves the assertion on boundedness. ç now prove the assertion on compactness. Note that As_(a+2)/p and A. are MSbius invariant, in the sense that every composition operator induced bv a conformal automorphism of the unit disk lna.ps each space into it.self, and contained COMPOSITION OPERATOlqS ON SOBOLEV SPACES 2841 in the disk algebra of holomorl»hic fimctions on the unit disk that extend tobe continuous on the closed disk. Thus a general theorem of 3. H. Shapiro [Sh] asserts that compactness of C' on each of these spaçes implies that ç(D) is a relatively compact, suhset, of D. We recall also that If(0)l + ,,p{( -]:loe)t-lf'(z)l z D} is an equivalent norm on Az; sec Thcorem 4.1 in [CM]. Now, let {f} lw a l»oundcd sequence in A.«-(a+2)/v- must show that some subsequence of {f,ç} converges in A-(+2)/v- know tllat {./} is a normal falnily, and thus a sut)sequence (which we still call {f}) converges to some f G H(D) tmiformly on compact subsets of D. Also, if C is compa.ct on A,, then it 4 v C A-(a+2)/p. HellCe (1 --I=l=)-+(+=)/vl'(z)l i ll, liforllllv bO, ll,ded o11 D. and it follows that I (0) - (0)l + (1 -Izloe)l-++oe)/l () - ,f' ()ll'()l uniformly on D as n , sincc ç(D) is c'ontaincd in a compact subsct of B. This means that {f, o F} c«mvcrgcs to thc fim('tion 9 = .f o ç in A-(a+2)/v, and so C A-(a+2)/v A-(a+2)/v is compact. Thc 1)roof Criteria fi)r C to lc 1)omded or compact on A: are known. So Thcorem 3.2 can be used to provide necessary condit.ions for boun(ledness or compactness of Cç on the smallcr spaces. In particular, we recall that the bomdedncss on A: implics the existence of the angular derivative of ç at all points of the unit circle whcre ç has a. radial limit of modulus 1: ste Corollarv 4.10 in [CM]. This l)roves the following theorem. Theorem 3.3. Let 1) > O. > -1 atd + < p P , is bouned, the the a9ular derivtttive of ç exists af all points Ç OD where the ro.dial limit ç(ç) of ç ezist.s a,,d so.ti.sfies I(çl = 1. As mentioned in the introduction, the conclusion of Theorem 3.3 is false for = -1, s = 1 and p = 1. Thus, for = -1, the lower bouud 1/p for s camaot be decreased in general. also give an example which shows that the lower bound s > a+ in Theorem 3.3 is sharp in case o > -1. See Examl)le 6.3 below. The proof of the next theorem is based on Theorem 2.6. So, for simt)licity, we restrict our consideration to the orders of differentiat.ion covered there. Theorem 3.4. Let p > O. a > -1. s > 0 ad p (a) If 1 + + s > e > 0 a'nd C is bouded (compact. resp.) on A,+«. tbe p so is C on A,. Proq[. Let I be an arc in the unit crcle, and let ç(z) = w G SI. A standard argument, using the Schwarz Lemma then tells us that 1 -]z 1 -[w] I[, and SO w SI. Hence Iz,a+«+(-)v(SI) Ille#,+(i_,)v(SI) 2842 ]3. CHOE, H. KOO, AND Ve. SMITH and statelnent (b) is now an imluediate consequence of Theorem 2.6. The proof of (a) is similar alld will be onfitted. [] We finish this section by giving the proof of Theorem 1.4 from the iltroduction. which we rest.ate for convenience. Theorem 3.5. Let p > O, o >_ -1, s >_ 0 and aasume ps = a + 2. (a) If C " AVa,s - A, s is bousd«d, thes C " VMOA - VMOA is bounded. (b) If ç is univalent and C " A, s -- 4v,s is compact, then C VlklOA -- VIklOA is compact. III the proof below and elsewhere, we use the notation dist(a. OE) for the Eu- clidean distance between a point a and the bounda.ry of a set E. Proof. If C is bolu,ded Oll .4P,s, then ffoin (1.6) we bave that ç = Cz 4P,s C VlklOA. Also, it is easy to see that (7, is bomlded on VlklOA if and onlv if ç Vlk[OA: see, for exalnple, [Sm2]. This gives part (a). For the proof of (b), we recall that when ç is Ulfivalent. C is COlnpact on VMOA if and Olfly if dist (w, 0ç(D)) \(Dl(W) (3.4) lira = 0: ,,,-+,- (1- [wl) see Theoreln 4.1 in [Sln2]. Also, it is ait easv cousequence of the Koebe distortion theol'em that if ç is univalent, then (3.5) ( -l[2)l'(=)[ dist(ç(z),Oç(D)), z D: see Corollary 1.4 iii [P]. First, consider the case p > 1 and c > -1. With ps = e+ 2 < p+ct + 1, case (i) of Theoreln 2.6 (b) tells us that C is COlnpact on AP,» if and only if/xp,p_2(SI) = o(1I[ ) as [I[ 0. We prove part (b) by showing that this fails when C is hot compact on VMOA. From (3.4), if (7 is hot compact on VMOA, then there is an g > 0 and a seqllence {Wh} C (D) with [zt,,[ -- 1 and dist(w,,0ç(D)) > e(1 -Iwl). Let I,, be the arc of the unit cirçle with ceuter u,,/[w,[ and length [I,[ = 2(1 -[w,[). Since ç is univalent. L {Iç'(-')l(1 Izl2)F-2d.4(w), ]'P'P-2(Sln) = In -- where w = ç0(z). Froln (3.5), Iç'(z)l(1 -Iz[ 2) (1 -Iw,l) for w in the disk with center w, and radius e(1 -Iwl)/2. which yields the lower bound (s) # o(llI"), III - 0. as desired. Hence [Lp,p_ 2 Now, consider the general case p > 0, c _> -1 and suppose that is compacL With ps = + 2, choose q as in the proof of Theorem 3.2. That is, choose q > p so large that q > 1 and put fl : sq- 2 > -1. Then +2 _ +2 = s, q p and so A, s C A, s by (1.4). Thus. froln Theorem 3.1 we see that C. " A, s A, s is compact and thus so is C VMOA VMOA by the result for the special case that we have proved above. The proof is complete. (-'OMPOSITION OPERATORS ON SOBOLEV SPACES 2843 4. COMPOSITION OPERATORS ON AP,s In this section we prove Theorems 1.1 and 1.2 flore the introduction. For con- venience, we divide these results into more easilv managed pieces. As meutioned in the introduction, it is well known that every composition operator is bounded . 4 p bv (1.1), on A for ail p > 0 and > -1 Note that we have A, when sp < + 1. Thus, it follows that cvery composition operator is boundcd on A, s whenever sp < + 1. Thc next two theorems complete the description of the general situation, as statcd in Thcoron 1.1. Theorem 4.1. Letp>O, s>0 ad >-1. Ifs- a+l atd (a) p > 9 or = -1 then every composilimt operator is bonded on 4 v ; (b) p < 2 and a > -1. then some composition operators are uot bounded on Pro@ If = -1. then s = 0 and so every composition ol)erator is b(mnded on A, = H v. If a > -1. p 2 and ps = a + 1. then froln (1.3) we have that H p C A,« Hence part (a) follows ff'oto Thcorem 1.3. since all composition operators are bounded on H p. a+l Turning to thc proof of (b), firsI note that A, s Ap_l, 1 p P P is necessarv for @ to be bounded on Ap_l. 1. Thus it Also, ç = @z ¢ Ap_l, 1 . suces to show that if p < 2 there is a bounded analytic function F Ap_13. The case p = 1 of this statemeut is outlined in exercise 9(a) iii Chapter VI of [G]. That construction can be modified to work for p < 2. For completeness, we sketch the argument. Let p < 2 and consid«r the flmction f(z) = k-llçz 2. Since the series for f is lacunary with square summable coecients, it is known that f BMOA. This is an easy consequence of BMOA being the dual of H 1 together with Paley's Inequality for the coccients of an H 1 flmction ([D], p. 104). or see [Mi] for another approach to the proof. Next, it is easy to verify that if :¢A,,.={wcD" 1-2 - then [ff(z)[ -l/P2. This leads to the approximation £ if,(z)lp( 1 _ izl2F«dA(z ) 1 flore which we see that f Av_I. 1. This is hot the l'equired example, however. since f is hot bounded. But since f ¢ BMOA, there are bounded flmctions Ul and u2 on the unit circle such that ef = Ul + fi2 where ri2 denotes the harmonic conjugate of u2. Here, we are using the saine notation for a boundary flmction and its harlnOlfiC extension. Then If'lp IVu I p + Iv.u21 p by the Cauchy-Riemama uations, and it follows that there is a bounded real flmction u on the circle such that L iv,()i(1- Now let F = exp(u + i'ù), so that F Ap_,, and the lzl)P- dA(z) = oo. is a bounded analytic flmction satisfying proof is complete. [] 2844 I3. CHOE, H. KOO, AND A. SMITH c+l The proof of the next theorem, covering the case s > ---, reqnires two lemmas, which will a.lso be used in the next section. Lelnnla 4.2. Let p > O. s O. and a -1. Then the following inclusions hold: (a) A,. c A ç for e > 0: (b) A . for O < < p. Ap+a_e,l+ s C ., Moreover, both iwlu.s'ions are bomMed. We remark in I)assing lhat, when s = e = 0. a = --1 and p 2. the inclusion iu (a.) holds and this is just a restatement of the well-known Littlewood-Paley inequalily. When s = e = 0, o = -1 and 1 < p 2, the inclusion in (b) holds and lhis is a restalement of the (lual of the Litllewnod-Paley inequalily. Proof. Bv definition of the holomorphic Sobolev spaces, it is suflïcient to consider the case s = 0. First. consider lhe case o > -1. Then, xve have il lld .4 c 4 ' , P o+e Ap+a+e,l, > 0 4 v " A; 4 p APo, 0 < e < p " p+o--e,1 o,e/p C. oe+e,e/p where the eqnivalences are ri'oto (1.1) and lhe inçlusions are clearlv bounded. Now, assmne (t = -l. Let J'G H p and put 1 (4.) .x(f.,.) = . If(,.d°)FdO. o < < 1. Then, for anv > 0. we have Mp(f.r) = O ((1 - r)-), which implies Mp(f',r) = O ((1 - r) -t-) (see Theorem 5.5 of [D]). Thus. for e > 0. integralion using polar coordinates shows lhat fb If'(z)lP(1- 'z]2)P-l+edA(z) '(1- F)--(1+6)P+P-I+dF. With small enough so that pO < e, this integral is convergent, and so (a) holds. Now, assume 0 < ¢ < p. Çonsider the case p > 1 first. Let p' be the conjugate exI)oncnt of p. Bv the fundamental theorem of calculus and H61der's inequality, we have ,{1 [, II(e '°) - I(0)F"0 ,. II'(rd°)[& eO JD xvhere It follows that I fo I , "1 p/p' C = (1 - r) -p (P-a-)/Pdr < IIf[Iï4,, £ [f(o)l p +/D If'(:)lP(1 -- COMPOSITION OPERATORS ON SOBOLEV SPACES 2845 and the Saille is true for p = 1 by a trivial lllodification of this argunlent. This proves (b) fol" p _> 1 by Proposition 2.2. When 0 < p < 1, we bave tlle inclusions Ap/(l-e) _ Hp/(1-e) Hp A;_ l_e, 1 C -1,0 -- C wheïe the fiïst inclusion conles ri'oin (1.4). This colnpletes the proof. [] We next show that certain inçhlsions t)etwoen hololnOl-l»hic Sobolev spa«es are conlpact. Lemma 4.3. Let p > O. s >_ O, ¢ > 0 and c >_ -1. Then the followin 9 itclusions are compact: p 4v A v A,,s+e C C Proof. We first consider the case > -1. Let {f} be 4 p is compact, we nmst show that A, s. To show that the inclusion A,, C . some subsequence of {f,} converges in 4 p I1 is well known tlmt the inclusion P P is compact: ste, for examplc [Sml]. Thus there exists g ,4 C . subsequenc of {f } (whic for convenienc we continue t o denote {f })suc that sfn g ill .4 pt+e. Now. chooscb (D) Sllch that h = g. It is then clear that 4 p This completes the proof that ,4 s C 4 p is h Ç 4P +e,s and f h in . ,+e,s. , compact. Now, since we have by (1.1), 4 , p Ap " a,s+e C Aa+pe,s+e . and the first inclusion is compact, we con«huh' lhe compactness of the inclusion A p Now, assulne = -1. Choose positive nunibers 1. 2 such that 0= < a < min(p, pe). Then, we have the relations 4 v v v A v Al,s+ e C « p-l+pe-6,l+s+e Ap-l-6A+s Q Ap-l-6,l+s The first, and la.st inclusions are bounded froni Lemma 4.2. the equivalence is ffOln (1.1), while the relnaining inchlsion is compact hy the previously established part of this lemma. Hence AP -1,s+e C AP -1,s is çOlnimct. For the compactness of A p -l.s C A p noe that -l+e,s 4 p A p A p 4 p -ls C p-l+e/2.1+s -l+e/2,s C -l+e,s" The first inclusion is bounded from Lemma 4.2. the isomorphism ri'oto (1.1), while the renmining inclusion is compact bv a previously established palt of this lemlna for a > -1. Hence AP -1,s C AP -+e,s is compact. The proof is complete. Theorem 4.4. Let p > 0 and o operators are hot bounded on AP,,. > -1. Ifs > +1 -- p then some composition Proof. Since a = Csoz Ç A, s is necessarv, for C to be bounded on -4P,s, it suflïces to show that if s > +__1, then H \ AP, -J= 0, where H denotes the class of ail p bounded hololnorphic filnctions on D. Suppose to the contrary that H C APo.,. Then this inclusion map is continuous bv the Closed Graph Theoreln. while the inclusion map AP,s C Aï.s was shown to be COlnpact whclmver a < /3 in Lemma 4.3. The hypothesized lower bound for s now allows us to choose _> 1 such that c + < .sp <_ c + + 1. Moreover, we can choose « > 0 so Sllmll tllat 2846 B. CHOE, H. KOO, AND W. SMITH P HP/(«+a+l+«-sp) from (1.4). 0 < c + 6 + 1 + e - sp < 1, which gives Aa+a_l+e, , C Consequently, we bave a chain of inclusions tt C APa C A p C tt p/(a++l+e-sp) , a+-- 1-t-e,s " Thus the inc|usion H'° C H p/(a+a+l+e-sp) can be viewed as a product of a compact nmp and two bounded maps, and hence is compact. But {z } is a bounded sequence iu H for which no subsequence converges in H Pl(a+'++e-p). This contradicts the COlnpactness of the illclusion lnap, aud the proof is colnplete. [] The next theoreln (also stated as Theorem 1.2 in the introduction) shows that the upper bounds for s in Theorens 4.1 and 4.4 can be extended whell the symbol ç' of the COlnposition operator is of boumled valence. Theoreln 4.5. Let (, > -1 and let ç be of bounded valence. Assume that either a+l 0_< s _< «+--v if p_>2. or'O_< s < --K- +½ if O < p < 2. Then Co zs bounded on Pro@ \Ve use the Carleson measure criteria frOlll Theoreln 2.6. We need to esti- llla[ e P,a+(1-s)p(qI)=i Z I'(z)]P-2(1-]z]2)«+O-S)PdA(w) o(z)=w for arbitrary arcs I C OD. First., consider the case p > 2. By aSSUluption we have sp _< a + 2. Note that, since ç is of bounded valence, there is a Ulfifornfly bounded lmmber of terlns in the sure inside the integra| above. Next. we set w = ç(z) and use the Schwarz-Pick Lemlna, which asserts that ]ç'(z)] < (1- ]w[2)/(1- [z]2), and then the elelnentary inequality 1 -[zl 2 <_ C(1 -[wl 2) to get that which from Theorem 2.6 is equivalent to boundedness of C on APa, . We uote that the hypothesis p _> 2 was used in getting the first inequa|ity, and sp _< c + 2 was used iii the second inequa|ity. Now, consider the case p < 2. What we have now is sp < c + 1 + -. By the area formula (Theorem 2.32 iu [C[]), we have f IAP,a+(1-s)P° (SI) = Jsal-' (si) Iç'(z)[P(1 -Izl2)+O->dA(z) Note that, since ç is of bounded valecce, we bave Æ I'(z)[2dA(z) = £ Z l dA(w) £ I,I 2 -(s,} , :e_{} COMPOSITION OPERATORS ON SOBOLEV SPACES 2847 where the first ilmquality is provided by H/31der's inequality. To estilnate the integral above, recall that everv composition operator C is bounded on A, 13 > -1, and that this is equivalent to (4.3) __ £_1(SI)( 1 __ iz[Z)ZdA(z) for all I C 0D: sec section 4 in [hIS]. Silwe, by hypothesis, 2( + (1 - s)p)/(2 - p) > --1, we Call combine the estinmtes iii (4.2) and (4.3) to get that lZ,a+(l_s)p(i) From Theorem 2.6. this is equivalent to C being bounded on A. s. complete. The proof is [] Our final result in this section shows that the conlpactness of the inclusions in Lemma 4.3 extends to ail colnposition operators with a certain restriction on s. Theorem 4.6. Let p > O, a >_ -1 and s > O. A,ssume that either (i) o. > -1, 0 <_ s < c+___ilp or (ii) c = -1. s = 0 or (iii) = c+.__!p .p>_2. (a) (Tv A,,+« -- Ag,, is compact. (b) c Aa_e, * -- APa,, is compact whenever ce - e -1. . 4 Pro@ We may view the action of Co - a,s+e -- APa,, as follows: p Ap c - A,. Aa,s+ e C a,s Also, if o - e > -1 then we mav view the action of C A p .4 s as follows: p C 4p A_«,, c A, The ilmlusions above are both compact by Lemma 4.3. and everv composition operator is bounded on A, by Theoreln 1.1. Thus, both operators m'e compact. The proof is complete. Carleson lneasure criteria for Co to be bounded or COlnpact Oll Hï are known; see Theorems 4.11 and 4.12 in [CM]. For s hot an integer, characterizing when Cso is bounded or compact on H seems much harder than the analogous problems on the Berglnan spaces. The problem is that for the Hardy spaces, (1.1) does hot provide isomorphisms with spaces defined using fldl derivatives. Thus. we are led to the following. Problem. Characterize ç for which C is bounded (compact) 011 HPs , 8 > O. 2848 B. CHOE, H. KOO. AND W. SMITH 5. (',OMPOSITION WlTH A POLYGONAL ,IAP Here. xve find simple criteria for the COlnpactness of the composition operators betv«een holomorphic Sobolev spaces induced by polygonal lnaps. Recall that dist(a.0E) denotes the Euclidean distance between a point a and the boundary of a set E. Lelllllla 5.1. Suppose that P is a polygon inscribed in the unit circle with a verrez al v and let rq(t,) be the vertex angle al v. Then. given a Riemann map of D onto P. there es:.isls a neighborhood N of v such that I¢(z)l ( -Iqo(z)l) -1/'(), ( -14) (1 -Iq(z)l) /)-I dist(v)(z). OP) for all z ç-(N). Proof. Reca.ll that ç extends to a holneOlUorphisn of D outo P (see, for exalnple, Theoreln 14.19 of [R2]). Assulne v = 1 aud ç(1) = 1 for simplicity. Also, let q = q(v). Then, a reflection argmnent yiehis l l (): C ( 1 -- Z ) + 0(I I I +',) for SOlllt, constant c ¢ 0 and for ail z near 1. Thus, we have Iv/()l I: - l ": I: -v)(z)l -I/ (1 -Iv)(z)l) for z near 1. The last equivalence in the disploEv above holds, because ç(z) is contained in a lontangential region with the vertex at 1. This proves the first equivalence of the lemlna. The second equivalence is now a consequence of the estimates (11 _]z[)]ç'(z)]_< dist(ç(z).OP)<_(1-[z])lç'(z)[, which hold since ç is mfivalent: see Corollary 1.4 of [PI. The proof is complete. [] Lemma 5.2. Let P be a pollgon inscribed in the unit circle. Assume b > -1 and a + b > -2. Then, there exists a constant C > 0 such that /oe 1-lwl) a dist(w, iï)P)bdA(w) <_ CIII çS I for all arcs I C OD. Pro@ Let us introduce a temporary notatiolL For an arc I C 0D with center at ç 0D and III = 25, we let S5(ç) = SI. Assmne that is sufficiently small and P çl S5(ç) ¢ . Then. there is a constant C. depending only on P, such that S5(ç) c Sc5(v) for some vertex v of P. Assume t, = 1 for simplicity. Assuming that is sufficientlv small so that Se, e(1) contains no vertex of P other than 1. note that 1 -lu, I Il - u,] for w G PO Sc6(1). Now. (OlklPOSITION OPERATOIRS ON SOBOLEV SPACES 2849 we bave (1 -[w]) dist(tv, OP)bdA(w) constant depelMing ouly on P. The estiluate for large [] Recall that D(z. 1/2) deuotes tlw pseudohyi»erl»olic disk. Let D(z) = D(z. 1/2). In the following we let dA,(z) = (1 -Iz]=)dA(z) for c > -1. The following lenmm is proved fol" a = 0 in IL2], and the saine proof works for general a. Lemma 5.3. Let c > -I and It be a positive finite Borel measure on D. Assume p > q > (I. Then, there is a constant C such that if and onlv if r G L(A) where r(z) = u(D()) A,,(D(z))" For a polygon P inscribed in the uuit circle, recall that O(P) denotes 1/ times the measure of the largest vertex angle of P. Proposition 5.4. Let pi > O. ai > -1. si >_ 0 (j = 1.2) and assume O1 3-2 1 a2+l (5.1) s < 1 + , s < -- Pi P P2 Let be a holomorphic function takin9 D into a pol19on P inscribed in the unit circle. If pl (O2 -[- 2 -- 82102) (5.2) 0(P) < P2(al + 2 -- SlPl)' 4 1 -- A, , is bounded. lhen C¢ " Ol,Sl 2, 2 Mo.reover, for functions ç of bounded valence, the second part of (5.1) can be replaced by the weaker condition that c + 2 c2 + 1 1 (5.3) s2 _< -- if p2 >_ 2, or s2 < -- + if 0 < P2 < 2. P2 P2 - h either case the eq,uality tan be allowed in (5.2) for P2 >_ Pl. 2850 B. CHOE. H. KOO, AND '. SMITH Pro@ Let ç0 be a Rielnann nlapping of D ollto P all([ put ¢ = ç- o ç. Then ç = ç0 o and Oms C¢ = Ç¢oO = Cç'C¢o. Note that CO " 4 pz 4 p is bomded by Throrem 1.I or Theorem 1.2. This shows that we only need to prove the proposition for ç = ç0- So, in the rest of the proof, we assume that ç is a iemann map of D onto P. For simplicity, let = pj + j - sjpj and let ?=2+j-jpj forj=l,2. First, consider the case p2 OE Pl- By Theorem 2.6, ve need to show that (5.4) t'p, (SI) = 0([II (2+)p/m+(1-)p) for ail arcs I. As in the proof of Lemnm 5.2, we onlv need to considcr I centered at a vcrtcx of P for which ]I] is sufficiontly small. Given such I, we havc bv Lemma 5.1 and Lomma 5.2, , ]'(-i(w))]P-2( 1 -]-1 (u,)]2)OdA(u,) [ dist(w,0P)u(1 -]w[)(/°-i)ud.4(w ) IP where 0 = O(P). In he last inequality we used the fact that ff2 > -1, 2 0 from (5.3) and tlms Thus, we have (0.4) by (5.2) and (5.5). Also, the smne proof works in case the equality holds in (5.2). Next, consider the case p < p. lnav, assume ç(0) = 0. Let f ,4 m, be an arbitrary fimction such that f(0) = 0. Since p < p, we have > -1 bv the first part of (5.1). Thus, by (1.1) and Proposition 2.2, we lmve [[f][A, [ f'(w)[m(1 --[w[)dA(w) Also, we have by (1.1), Proposition 2.2 and Lemma 5.1, Now, define mesures dpa(w) = (1 -[l)dA(), and let r(z) = p2(D(z)) z D. pl(D(z))" COMPOSITION OPERATORS ON SOBOLEV SPACES 2851 By Lelmna 5.3, we need to show that v E LP(p) where p = Pl/(Pl - P2). Note that/t2(D(z)) = 0 if z is outside of some t)olygonal region Q. ()n the other hand, for z G Q. we have p,(D(z)) (1 -Izl) »l+, pee(D()) (1 -Izl2) +(/e-1)+ee', the first esçimate is standard and the second one çall be verified with (5.6) bv modiÇving the proof of Lomlna 5.2. According]y, we lmve (=) ( -i=l)=-»,+,/e-,,b( ). [t fo]lows that r e L(pt) if and on]y if p[fl2 - fl + (1/8 - 1)72] + fit > -2, which urns out to be thc saine as (5.2). his completes the proof. Theorem 5.5. Lct pj > O. ej -1, si 0 (j = 1,2) and assume (5.1) holds. Let ç b« a holomorphic function taking D into a polflgon P iscrih«d in th¢ unit circle. If (5.2) holds, fh«n Ç" 4 m 4 p= is compact. Mor«ov«r. for flmctio,s ç of bovnded valence, t« second part of (5.1) can be r«plac«d by the w«aker co,ditio** (5.3). In Examplc 6.1 below, we show that (5.3) providcs the sharp upper bound of +2 for s2 when p2 > 2. Whilc we do hot know whcther it docs the saine when P2 ( 2, the ni)per bomd for s2 when p2 ( 2 cannot be extended to :+ as is P2 shown by Example 6.2. Nevcrthelcss, Examplc6.4 shows the upper bomd of in (5.2) is sharp in eithcr case. Pro@ Assume that (5.2) hokls and choosc ¢ > 0 suciently small so that (5.2) holds with a + e in place of Bv Lemma 4.3 we have 4 p Ç 4 P and . 4 m 4P2 the inclusion is compact. Thus, it is sufficicnt to show that C is bounded. In case au > -1. we sec that C¢ A',+«, -4Pa,s is bounded bv Proposition 5.4. So. assume 2 = -1. Note that with a2 = -1 there is no s2 satisfying the second part of (5.1). Thus. we only need to be concerned about the case where is of bomded valence and (5.3) holds. First. consider the case p2 2. In this case, we can view the action of @ as follows: Ara cv p 4p a+e,s Ap2-1,se+l Ç ' -1,s where @ Ama+e,s APp-,s+ is bomded bv. ProI)osition 5.4 and the inelusion cornes from (1.2). Therefore, @ A m 4 p is boCmded. Next, consider lE,81 " 2,8 2 +2 _ çhecasep2>2. Choosepe(2,p2) ande >-1. Also, leçs= +s2 . Then. we tan view the action of C follows: 4m c ApÇ.s 4p where C " A m +e,s A,s; is bounded by Proposition 5.4 and the inclusion cornes from (1.4). The proof is complete. Remarks. 1. As mentioned in t.he proof above, t.here is no s2 satisfying the second part of (5.1) in case a2 = -1. Thus, we have no conchlsion in Theorem 5.5 for general ç in ce the target space is a Hardy-Sobolev space. 2. Note that the condition (5.2) holds vacuously if a+2 +2 P P2 2852 B. CHOE, H. KOO, AND ,V. SMITH 6. EXAMPLES rVe llOW give sevcral exmnples delnonstrating that our theorelns are sharp. For tiret purpose we introduce the so-called lens maps. Fol" 0 < 11 < 1 we denote by çn the flmction defined bv er(z) ' - 1 (6.1) qa,(z) -- er(z)' + 1" z C D where er(z) = (1 + z)/(1 - z). Let n(D) = L. Then, çn is the Rielnann map of D onto the sui»set L of D bounded by arcs of circles lneeting at z = +1 at an angle of lift, and fixing the points -1, 0, and 1. Because of the shape of the range L,» such a map is called a "lens map". Note that L is contailmd in a polygon inscribed in the unit cir«le. By a straightforward calculation, we bave (6.'-,) -I()1 1 - zl ', I'(z)l I] - zl for z near 1. The first example shows that the upper bomul s < +___2 in Theorem 1.2(a) is p slmrp. Also, this exalnple shows that thc upI)cr bound so < az+ in Theoreln 5.5 is sharp when p2 k 2 and is of bounded valence. Example 6.1. Let 1 »> 1, o >-1 and + < s < 1 + a+l Thon, thereexists a P P P lens map n -4,s" In particular, C is hot bounded on Aa, s. Proof. Choose 0 < 1 < 1 sucientlv small so that sp k IP + a + 2 and consider the corresponding lens lllap ç Note that A A P bv (1 1). Therefore, " , a+(l--s)p,1 - " we bave bv Proposition 2.2 and (6.2). ./ I'o (z)lp(1 -Izlu)+(1-PdA(z) aD Note that (r/- 1)p + er + ( 1 - s)p <_ -2. because sp _> 'lP + er + 9. Thus, an obvious estinmte iii an angle with vertex at 1 shows tiret the last integral above diverges, as desired. [] 1 We do hot know whether the upper bound s < a+__! + 7 in Theoreln 1.2(b) is sharp. However. the next exalnple shows t.hat the upper bound cmmot be extended 1 to + as in Theorem 1.2(a.). Also. this is related to the assumption, s., < =+ + , p p2 in Theorem 5.5 when P2 < 2. Example 6.2. For each p [1,2), there exist > -1, 0 < s < 1 with s < + p and a univalent holoInorphic self-map of D such tiret A p In particular, C is hot bounded on A,,. Pro@ P..lones and N. Makarov have showll (see Theorem D(2) in [aMI) that for any p < 2, there exist a univalent holomorphic self-map çp of D and a constant ' satisD c > 0 such that the integral ieans of p . P--I+c(2--P) P __ (6.3) (1 - ,') - I (wp- ") > 1 COMPOSITION OPERATORS ON SOBOLEV SPACES 2853 for some sequence 7", -- 1. Here, we are using the notation introduced in (4.1). Note that, for anv f E. a, /3 > -1, we bave a.ild so 1 (1 - r)+SIP(f,r)= (/3+ 1)/lç(f, r) .f (1 -t)dt < (/3+ 1) lllP(f,t)(1-t)13dt=o(1) ! as r 1. This, together with (6.3), yields çp tg Ap_2+c(2_p)2. hl other words, we have çp tg Ap_2+c(2_p)2,1 p bv. Proposition 2.2. N-te that thc hvpohesis p _> 1 is used here to ass,re that p - 2 + c(2 - p) > - 1. Now, «hoose s E [0, 1) su«h that sp - 2 + c(2 - p)Z > -1 and put « = sp - 2 + c(2 - p)2. Th('u, wc bave s < +2 Also. since 4 p p-2+c(2-p)=,l "-4Pa,s bv. (1.1), we havc çp 4 v,s. [] The next examplc shows that the lower bomd s > +-- in Thcorem 3.3 is sharp when o > -1. Exanlple 6.3. Let p > 1, ,. > -1 and put s -- +2 Then. there exists a P holomorphi« self-nmp ç of D with ç(1) = 1 su«h that C' APo, s -- APa,, is b(mnd('d but ç does hot have angula.r derivative a.t z = 1. Pro@ Let ç = çn be any lens lnap. Note that a + (1 - s)p > -1. Thus, as in the proof of Proposition 5.4, we have m that @ " A, A,, is bo,mded bv Theorem 2.60). Clearly, ç does hot have an angular dcrivative a.t 1. The next example shows that the upper bound for O(P) in Theorem 5.5 is sharp when Cil > -- 1. Example 6.4. O I > --1 and Let pi, si, aj be as in the hypotheses of Theorem 5.5. Assume Then, f o çu 4 2 PI (02 + 2 -- (6.4) p2(o + 2 - SlPl) < ri < 1. for some f G 4 P Cq ,S I " Pro@ Let ç = ç,. Choose 0 < a < 1 such that ]ç(a)l _> 1/2. Also. by using (6.4), choose « > 0 sufficiently small so that (6.5) (2 + o2)/p 2 -- .» + a - <]<1 (2 + O1)/p -- S Now, consider the test fimction f(z) = log(1 -ç(a)z). Let k _> s be a positive integer. Then we have 4 pI A pi « C,,I..S1 tl_]_(__S1)p,] by (1.1). Therefore, bv Proposition 2854 B. CHOE, H. KOO, AND W. SMITH 2.2, (2.3), (6.2) and (6.5), (6.6) < On the other hand, for e2 > -1, and thus by (6.2), (6.7) (1 - I(a)l 2 ) [BB] [CM] [D] [HKZ] [Ma] [PI [Pd] R EFERENCES F. Beatrous and J. Burbea, Holomorphic Sobolev spaces on the ball. Dissertationes Math- ematicae, CCLXXVI (1989). 1-57. MR 90k:32010 C. Çowen and B. IklacCluer, Composition operators on spaces of analytic functions, CRC Press. Boca Raton. FL, 1995. MR 97i:47056 P. Duren. Theory of H p spaces, Pure and Appl. Math., Vol. 38. Academic Press, New York. 1970. MR 42:3552 J. Garnett. Bounded analytic functions, Pure and Appl. Math., vol. 96, Academic Press, New York, 1981. lkIR 83g:30037 P. Jones and N. lklakarov, Density properties of harmonic measure, Armais of Mathemath- ics, 142 (1995), 427-455. lkIR 96k:3002ï H. Hedenmalm, B. Korenblum and N. Zhu. Theory of Bergman spaces, Graduate Texts in Math., vol. 199, Springer. New York, 2000. MR 2001c:46043 D. Luecking, Forward and Reverse Carleson inequalities for functions in Bergman spaces and their derivatives, Amer. J. Math., 107 (1985), 85-111. MR 86g:30002 D. Luecking, Multipliers of Bergman spaces into Lebesgue spaces, Proc. Edinburgh Math. Soc., 29 (1986), 125-131. MR 87e:46034 B. MacCuer and J. H. Shaprio, Angular derivatives and compact composition operators on the Hardy and Bergman spaces, Canad. J. Math., XXXVIII(4) (1986), MR 87h:47048 878-906. K. Madigan, Composition operators on analytic Lipschifz spaces. Proc. Amer. Math. Soc., 119 (1993), 465-473. MR 93k:47043 J. Miao, A property of analytic functions with Hadamard gaps, Bull. Austral. Math. Soc., 45 (1992), 105-112. MR 93c:30060 Ch. Pommerenke, Boundary behavior of conformal maps, Grundlehren der Mathema- tishen Wissenschaften, vol. 299, Springer-Verlag, Berlin. Heidelberg, New York, 1992. 95b:30008 W. Rudin, Funct,on theory in the unit ball of C n, Grundlehren der Mathematishen V'is- senschaften, vol. 241, Springer-Verlag, New York, 1980. MR 82i:32002 COMPOSITION OPEPATORS ON SOBOLEV SPACES 2855 [R2] [Sh] [Sml] [Sm2l [SYI [ST] W. Rudin. Real and complex analysis, McGraw-Hill, New York, 1987. M1R 88k:00002 J.H. Shapiro, Compact composition operators on spaces of boundary-regular holomorphic functions, Proc. Amer. Math. Soc., 100(1) (1987), 49-57. Ml 88c:47059 W. Smith. Composition operators between Bergman and Hardy spaces, Trans. Amer. Math. Soc., 348(6) (1996), 2331-2348. MR 96i:47056 W. Smith, Compactness of composition operators on BMOA, Proc. Amer. Math. Soc., 127(9) (1999}, 2715-2725. MIR 99m:47040 W. Smith and L. Yang, Composition operators that improve integrability on weighted Bergman spaces, Proc. Amer. hlath. Soc., 126(2) (1998), 411-420. hIIR 98d:47070 .1. Shapiro and P. Taylor. Compact, nuclear, and Hilbert-Schmidt composition operators on H 2, Indiana Univ. Math. J., 23(6) (1973), 471-496. MR 48:4816 DEPARTMENT OF h[ATHEMATICS, [OREA UNIVERSITY, SEOUL 136--701, [OREA E-mail address: choebr@math, korea, ac. kr DEPARTMENT OF I[ATHEMATICS, KOREA UNIVERSITY. SEOUL 136-701. KOREA E-mail address: koohw@math, korea, ac. kr DEPARTMENT OF ]IATHEMATICS, UNIVERSIT , OF HVAII. HONOLULU. HAV:«II 96822 E-mail address : wayne@math.hawaii.edu TRANSACTIONS OF TtiE AMERICAN MATHEMOETICAL SOCIETY "volume 355, Number 7, Pages 2857-2883 S 0002-9947(03)03248-3 Article electronically published on Match I9, 2003 DISTRIBUTIONS OF CORANK 1 AND THEIR CHARACTERISTIC VECTOR FIELDS B. JAKUDCZYK AND M. ZHITOMIRSKII ABSTRACT. oEre prove that any l-parameter family of corank 1 distributions (or Pfaff equations) on a compact manifold 5I ' is trivializable, i.e., transfornlable to a constant family by a fanily of diffeonmrphisms, if ail distributions of the family have the saine characteristic line field. The characteristic line field is a field of tangent lines which is invariantly assigned to a corank one distribution, If is defined on lI n, if , = 2/,', or on a subset of AI" called the Mmtinet tlypersurface, if n : 2k -I- 1. Our second main result states tllat if two corank one distributions have the saille characteristic line field and are close to each other, then they are equivalent via a diffeOlnorphisln. This holds tmder a weak assumption on thc singularities of the distributions. The second result ilnplies that the abnorlnal curves of a distributiol determine the equivalence class of the distribution, among distributions close to a given one. O. INTRODUCTION The well-known Gray theoreln [G] states that any 1-parameter fmnilv of contact structures on a compact manifold ,l).a+l is trivializable, i.e., transformable to a constant falnily by a fmnily of diffeomorl)hislnS. Our first main result generalizes this theorem to he case of 'singular contact structures", for which the contact condition is satisfied on a dense subset of M, and to cora|lk one distributions on llmnifolds of even dilnension. In these cases the famil.v of distributions has to preserve, when the parameter changes, a characteristic line fiel(l. The characteristic lie field is a field of tangem lines which is invariantly assigned to a corank one distribution (it is defined on M if = 2/,', or on a hypersm'face of/1I if = 2k+l). Our second main result states that if îwo corank Olle distritmtions have the Saille characteristic line field raid are close to each other, then they are equivalent via a diffeomorphism. It lneans, in particular, that the chm'acteristic line field contains COlnplete information about the geometry of singularities of the distribution. Our results hold under a weak &ssumption on the distributions, called conditiol saying that the depth of a characteristic ideal of the distribution is nondegenerate at singular points of the characteristic line field. Received by the editors January 9, 2002 and, in revised form, September 4, 2002. 2000 Mathematics Subject Classification. Primary 58A17; Secondarv 53B99. Key wovds and phrases. Pfaff equation, equivalence, contact structure, quasi-contact struct ure, singularity, invariants, line field, homotopy method. The first author was supported by the Committee for Scientific Research (KBN), Poland, grant 2P03A 03516. The second author was supported by the Fund for the Promotion of Research at the Technion. (2003 American 1Mathemalical Socict3 2858 B. JAKUBCZYK AND M. ZHITOMIRSKII Let us state out second result in the simple case where the characteristic line field does not have singularities. Assume = 2k >_ 4 and consider a smooth differential 1-forln ce on /1I n. Let (w A (dw)k-1)(p) 0 for ail p /1I n. Such a forln w defines the distribution A= kerce, called the quasi-contact structure defined by w. Then dw(p), ïestricted to the distribution Ap) = kerw(p), is of nmximal possible tank 2k - 2 aud has 1-dimensional keïnel. We define the characteristic line at p by L v = ker dw(p) The characteristic line fleld L = L(w) is the fiekl of characteristic lines p -=, L v on /11. The folloxving fact is a special case of our Theoreln 1.2, the case where singularities are absent. (It is also a special case of a theorem in [MZh], Appendix A, concerning corank 1 distributions of constallt class.) Theorem 0.1. Let A be tbe quasi-contact structure on a compact orietable man- ifold III 2 defined by a 1-form w. and let & be a 1-form such that L(&) = L(w). If & is sujïjïciently C2-close to ce, then there exists a diffeomorphism of M 2 sending ï = ker& fo A. If a compact orientable manifold ]I 2a adnfits a quasi-contact structure defined bv a global 1-form w, then its characteristic line field is generated by a nonvanish- ing global vector field and the Euler chmacteristic of/I 2a is eqnal to zero. Thus a manifold /I 2a with uonzero Euler dmracteristic adnfits only corank one distri- butious with singular characteristic line field. Eveu if/I 2a admits a quasi-contact structure, siugularities nmv appear naturally when restricting a corank one distri- bution to a subnmnifold of even dimensiou. Therefore, it is natural to ask whether Theorem 0.1 holds in the preseuce of singularities. Theorem 1.2 in Section 1 gives a positive answer under assumption (A), saying that the singularities of the char- acteristic vector field bave a natural dcpth (and codimensiou, in the analvtic case). This assumption excludes singularities of iufinite codimension. To state a siufilar result in the case n : 2k + 1. we introduce the set of points where w does hot satisfy the contact conditiou: S = {p IIl 2+ (w A (dw)k)(p = O}. This set is called the Martinet hypersurface. The Martinet hypersurface is the set of zeros of the flmctiou H = ce A where {2 is a volulne forln. If S is elnpty, i.e., ce is a contact 1-fonn on 3I 2+, then A = kerw is globally equivalent to any distribution , sufficiently close to A. This follows frolu the theoïem of Gray lnentioned above. Assume now that S is louempty. \Ve shall call A = kerce a Martinet distribution if it satisfies the following two conditions: (a) dH(p) ¢ 0 for all p S (then S is smooth), and (b) As = kerces is a quasi-contact structure on S, where ws is the pullback of wtoS. At each point of S we can define the characteristic line Lp = kerdws(P)]ks(p), p S. The characteristic line field L = L(w) for a Martinet distribution & is the field of tangent lines p -- Lp on S. It has no singularities. The following fact is a special case of out Theorem 1.4. DISTI:IIBUTIONS OF COI:IANK 1 2859 Theorem 0.2. Let A = kerw be a Martinet distribution on a compact orientable manifold 1112k+1 . If the Martinet hypersurfaces ad the characteristic line fields of A and/ = kera5 are the saine and & is sujïficietly Ca-close to w, t]en there exists a diffeornorphisrn of tle rnanifold sending A to A. Martinet distributions fonn a restrictive class of corank one distributions (L may have singularities). In particulm, any Martinet distribution on M 3 has the Mmtinet hypersurface S, which consists of two-dimcnsional tori (each connected component of S has zero Euler characteristic, since S admits a l-dimensional foliation defined by L). Out Theorem 1.4 will generalize Theorem 0.2 to thc case of glmral corank one distributions and PfaIf equations on/ll 2k+1. The appemauce of the characteristic line field L as one of the main invariants of corank one distributions A has the following history. The first to study L -cas J. Martinet in [Mari for the simplest occurring singularities of X on IR 2+ (when L has no siugularities). J. Martiuet also started to study typical singularities of L in the 3-dimensional case. These singularities were roughly classified in [JP], where the existence of a modulus in the classification of characteristic line fields was shown. It was proved in [Zhl] by obtaining a normal form for A that this modulus is the only iuvariaut of A. This gave thc complete local classification of germs of generic 2-distributious ou 3-manifolds, with the characteristic line field as the complete invariant. In the book [Zh2] the second author gave a classification of finitely determined singularities of corauk one distributious and PfaIf equations on mauifolds of any dimension. In this case again the chm'acteristic line field L is a complete invariaut. This justifies, to a large extent, lnaking the followiug Conjecture. In the space of germs at 0 IR ' of corank one distribations on IR ', there is an ezceptional set of infinite codirnension such that for any two distributions Ao and As away front this set, with the saine characteristic line field L, there ezists a local diffeomorphism op : (IR ', O) -- (IR ', O) redacing X to A0. In a weaker form, with n = 3, this conjecture has already appeared iii [JP] and in a letter from J. Mmtinet to the second author (in 1989). In this case the conjecture tan be deduced from the results in [JZh2], [JZhl]. The exceptional set consists of germs that do hot satisfy the assumption (A) or that do not have the property of zeros (sec Section 1). In [JZh2] we proved that away rioto the exceptional set the restriction of the distribution to the Martinet hypersurface is a complete invariant for any n = 2k + 1. If k = 1, then the restriction eau be identified with the characteristic line field. The results of the present paper concern global corank one distributions close to a fixed distribution. They also deal with families of distributions. Iii this setting we eliminate some of the diftàculties iii the above conjecture which are due to non-close germs and the necessity of preserving a fixed point (the source of the germ). Our transition to the global approach was inspired by the results in [Gol] and [MZh, Appendix AI. In [Gol] it is proved that a fmnily of Engel structures Et on a 4-manifold is trivializable provided that the characteristic line field of Et does not depend ou t. The result in Appendix A of [MZh] states that two close global corank one distributions of any constant class (iu the Cartau-Frobenius sense) are diffeomorphic provided that they have diffeomorphic characteristic foliations. This is a generalization of the Gray theorem. All these results apply to objects without 2860 B. JAKUBCZYK AND M ZHITOMIRSKII singularities, whereas in the present paper we allow any singularities except certain ones of infinite codimension, excluded by the assumption (A). The presence of singularities leads to the main ditïficulties iii ollr proofs. In Section 2 we explain tllat, the assumption (A) is natural and give examples showing that it caimot be weakened. In [JZhl] and [JZh2] we ohtained local realization theorems {for gerlns at a fixed point }, t heorems characterizing the set of all possible characteristic line fields if r = 3. Combined with the reduction theorems, thev lead to a number of applications including classification results. In this paper we leave aside the diflïcult task of obtailfing global realization theorems. The absence of such theorems restricts, at present, the possibility of drawing imlnediate iuteresting conclusions concerning global classification of corank one distrilmtions. Note that the assmnption of closeness which appears in Theorems 0.1 and 0.2 {and later in Theorelns 1.2 and 1.4} is essential for out results. Already two contact structures that are far ff'oto each other are, in general, hot equivalent. A classifica- tion of contact structures is known only on certain 3-diluensional manifolds, sec e.g. [TEG]. We hope that our results can be used in glolml contact or quasi-contact ge- »metrv for studying singularities of 1-forms which appear when two global contact (or quasi-contact) structures are joilmd by a path. There are natural consequences of our results concerning characteristic curves of a distrilmtion, also called sigular curves or abormal cu.rves in sub-Rielnannian geometly, all(l t]le geolnetry of distril,utions (cf. lAI, [BH], [LS], [Mon]). These curves coincide in out case with t]le integral curves of the characteristic line field. Under thc assumptions of Theorerns 0.1 ad 0.2, as well as those of Theorems 1.2 and 1. it Se«tiot 1. the singular curves of a «otanC" 1 distribution &termine the equivalece class of the distribution, amog ditributions close fo a gi-ven one it the C topolog. 1. TATEMENT OF tlESULTS We will deal with Pfaff equations, v«hich are more general objects than (coori- ented) corank one distributions. Let. M u &,note a compact, orientable, Hausdorff lnanifold of dimension n _> 3. Bv definition, a Pfaff equation is a set of differential l-ri»tins on M' generated, as a module over the ring of fimctions, bv a single 1-form w. In other v«ords, a Pfaff equation is a 1-form on .1I ' defined up to lnultiplica- tion bv a nonvanishing flmction. We dcnote the Pfaff equation by P = (w). If w vanishes at no points of M', then (w) can be identified with the field of kernels of w--a coorientable hyperI)lalm fie]d in TIIl '. In general, a Pfaff equation is a more general object, siuce we do not exclude the possibility of w vanishing at some points of the manifold. Ail objects in this paper will belong to a fixed category which is either C or real analytic C ". The case = 2/,'. To any Pfaff equation P. and in particular to any cooriented corank Olle distribution, one can associate the characteristic line field. Definition 1.1. If n is even. n = 2/,', then anv vector field X defined by the relation XJ = to A (dw) k-l, DISTF{IBUTIONS OF COIRANK 1 2861 where t is a.lV vohune form, a.ll(l ce any geuerator of P, is called a cbaracteristic vector field of P. The line field on In generated by X, i.e., the lnapping is called the «haracteristic l{e fie[d. Thc characteristic line field will be denoted by L or L(w) or L(P}. The set {p I X(p) = 0} is callcd thc set of singular points of L and dcnoted by Sing(L). It is easv to check that anv two characteristic vector fields differ bv multipli- cation by a nonvanishing fuucIi(m, and consequently thc characteristic line field is invariant.ly related to the Pfaff equation P, i.e.. the choice of the generator of P and the volmne form [ is irrelevant. Note that if ( A ()k-1)(p) 0. then the definitions of Lp and L coinçide with those given in the introduction. In the presence of singularities we need ail invariant that describes a "degree of degencration'" of singular points of a characteristic vector field X (for n odd il was introduced in [JZh2]). First we define i/il invariant that is slightly stronger than the set Sing(L). Let p Sittg(L ). Thc loc«d chwracteristic ideal Ip at a point p G .I 2k of a Pfaff equation P is the idcal Iv(X ) in the ring of functioli gerlns at p. gcnerated Iv the coefficients (i 1 ..... a,, of a characteristi« vector field X of P. in some cooMinate svstem near p. It is easv to sec that the ideal Ip is invarialitlv related to the gerln at p of P = () (the choices of a chara«teristic vector field and of a local coordinate system are irrelevant). The germ at p of the set Si,(L) is the zero set of Definition 1.2. If n = 2k and p Sittg(L), then wc define dp(P) = dp(X) as «(P) = d,,pth IdX). Recall that the depth of a proper ideal I C R of a ring R is the maximal length of a regular sequence of elements in I. A sequence ai ..... af I is called regular if al is hot a zero divisor in and. for any i = 2,..., r. the element ai is hot a zero divisor in the quotient ring R/(a,.... ai-l), where (al ..... (ri-l) denotes the ideal generated bv ai,..., (i-1. Bv definition, depth R = . Remark. In the alialvtic category, dp(P) is equal to the codimelIsion in Ç' of the germ at p of the set of complex zeros of the ideal Ip (i.e., the zero level set of the ideal generated by the complexification of the generators of Ip in some local coordilmtes). This follows ff'oto the fact that the complexification does hot change the depth of an idem of analytic fimctioli germs (cf. e.g. [E]) and ffoto the equality of the depth(I) and the codimension of the analvti« set of zeros of I for ara" ideal I of holomorphic function gel'lUS. XVe introduce the followilig crucial condition: (A) dp(P) 3 for anv point p Sing(L ). This condition is rather weak, in particular generic, as will be explained in Section 2. The following theorems hold in the cat.egories C and C , with AI a compact orientable manifold. Theorem 1.1. Let Pt. l [0, 1], be a famil of Pfaff equations ot ;I . k 2. that satisfies the followin9 coditions. (a) All P, define the saine characteristic line field L = L(Pt). 2862 13. JAKUBCZYK AND bi. ZHITOMIRSKII (b) Ail Pt satisfy condition (A). Then there exists a family (Pt of diffeomorphisms of ]il 2k sending Pt to Po. Theorem 1.2. Let Po = (Wo) and P1 = (wl) be Pfaff equations on ]il 2¢. k >_ 2, that have the same characteristic line field L = L(Po) = L(P). Assume that condition (A) holds for Po. Then there exists a diffeomorphism (P sending P to Po provided that w is suJficiently close to Wo in the C topology. In the above theorems as well as in Theorems 1.3 and 1.4 below, all objects are in the saine category C or C °, including regularity with respect to the paranmter t. The diffeomorphism in Theorem 1.2 can be taken Coe-close to the identity. Remark (Closeness of w to w0)). In nmny cases one can present a number r < OE (depending on Po) such that the closeness in the C topology in Theorem 1.2 tan be replaced by closeness in the Cr topology. See Theorem 0.1 and Theorem B.2 in Appendix B. The case n = 2k + 1. The most basic invariant of a Pfaff equation P = (w) on /I 2+ is the set S = {p e //12k+ : (w A (d)k)(p) = 0}, called the Martinet lypersurface, which consists of points at which w is not a contact form. This set, invariantly related to P, is the zero level of the function /4 = A (d)/f, where ç is a volulne form. The ideal (H) of the ring of functions on ]il 2k+, generated by H, is also invari- antly related to P. It is called the Martinet ideal. The characteristic line field of P = (w) on/12k+ is defined on the set S. Definition 1.3. Anv vector field X on/i2+ satisfying the relation X]i = o3 A (dw) k-1 A dH mod (H). where w is an 3, generator of P, and H is an3" generator of the Martinet ideal, will be called a characteristic vector field of P. The line field on S = {H = 0} defined by the relation p-- L» = {eX(p), a e OE}, p e S. is called the characteristic line field of P. It will be denoted by L or L(w) or L(P). The set of singular points of L is defined as Sing(L) = {p e M 2+1 H(p) = o, x(p) = 0}. Above and in the test of the paper the equality of two objects (vector fields, differential forms) mod (H) means that their difference is divisible over H in the space of objects of the saine category. To check that the line field defined above is tangent to S, note that the definition of X implies that XJdH = 0 at any point of S = {H = 0} and that X vanishes at any point of S at which the l-form dH vanishes. Thus X(p) Ç TpS at any point p S at which S is a slnooth hypersurface and X vanishes at all other points. It is easy to check that the characteristic line field is invariantly related to P, i.e., the choices of the generator w of P. the generator H of the Martinet ideM, and the volume form Q are irrelevant. Note that in the case of the Martinet singularity (p S and (, A (d,) k- A dH)(p) 7(= 0) the definitions of Lp and L coincide with those given in the introduc- tion. DISTIIBUTIONS OF CORANK 1 2863 In order to deal with deeper singularities of P, namely those allowing singular points of L, we introduce out invariant dp(P) in the case of odd n as follows. Let p E Sing(L). The local characteristic ideal Ip at a point p G /_/2k+l of a Pfaff equation P is the ideal Iu(H, X) in the ring of function germs at p generated bv the germ Hp of a generator of the Martinet ideal and the coeflïcients of a characteristic vector field X of P, in some coordilmte system near p. The ideal I u is invariantly related to the germ at p of P = (w) (the choices of a characteristic vector field, a generator of the lklm'tinet ideal, and of a local coordinate system are irrelevant). The germ at p of tlm set Sig(L) is the zero set of Ip. Definition 1.4. Il: = 2k + 1 and p G S«ug(L), then we define du(P) =du(H,-\" ) as the maximal length of a regular sequence in the characteristic ideal Ip(H. starting with the germ H u as the first element. Remarks. (a) In Noetherian rings all nmximal regular sequences in I are of the saine finite length, lk[oreover, any regular sequence can be completed to a maximal regular sequence. This implies that in the analytic category, independently of the parity of n, we have du(P) = depth(1 u). (b) Similarlv to the case = 2k, in the analytic category, du(P ) is equal to the codi- mension in C of the germ at p of the set of complex zeros of the complexification of the idem To formulate our reduction theorem for the most general case, we need two prop- erties of the Martinet ideal (H): the property of zeros and the extension property. Definition 1.5. The Martinet ideal (H) has the property of zeros if for any p S = {H :- 0} the ideal in the ring of ail function germs at p generated bv the germ Hp of H at p coincides with the ideal in the saine ring consisting of function germs vanishing on the germ at p of the set S --- {H -= 0}. The property of zeros allows us to identify the hlartinet hypersurface S 0} with the hla.rtinet ideal. In the case of germs this follows from the definition. Examples where the property of zeros is violated at a point p include: (a) H u = HîH2. where Hx, H2 are function germs and H(p) = (b) H u is equivlent to r 2 = xî +-.-+ x; (c) Hp is a fiat germ (i.e., the Taylor series of H at p is zero): (d) H u is a zero divisor in the ring of all germs at p. In case (c) the property of zeros is violated, since u = r-2Hu is smooth and has the same germ of zeros as H u but p ¢_ (Hp). Note that (d) is a particular case of (c). The local version of the property of zeros (Definition 1.5) implies the global version: if a function f on/I vanishes on the set S = {H --- 0, then f belongs to the ideal (H). This follows from the fact that division by H or by the germ H u is 1 unique (by (d) the germ H u is hot a zero divisor). In the Coe-category the global and local versions of the property of zeros are equivalent (this follows from the partition of unity). In the real analytic category they are equivalent provided that the sheaf of functions vanishing on S = {H = 0} s coherent. In the proof "global implies 2864 B. JAIxUBCZYK AND M. ZHITOMIRSKII We also need the e.rtension propert. of the Martinet hypersurface S = {H = 0}. Denote by Çoe(l) the Fréchet space of smooth fimctions on AI, equipped with the topology of convergence together wfih all derivaiives. Let Ç(,l. S) denoie its closed subsl)ace of functions that vanish on S. define the space of smooth funclions on S as the quotienl Déchet space C(S) = COE(kI)/Ç(:II. S). Definition 1.6. sa,y that S has the ea'tensio propert if there exists a COlltillU- o,,s linear operator A " Ç(S) ÇX(kl) such that A(f)[s : f for ail f G COe(S). The extension property automat.ically holds in the Ç category, since it holds for anv analytic subset S of Al (see [BS] tbr a more general extension theorem). It also holds if wo assmne that (H) h, locally aromid any point p G S. a generator that is analytic in some coordinate system. The fifllowing theorems hold in the categories Coe and Ç, with M compact and oriental)le. Theorem 1.3. Let Pt. [0.1], be a famil# of Pfa equatious on M +. k 1. that satzsfles the follouring conditions. (a) All Pt bave the .saine Mn«'tinet hyper.surface S, which bas the etesion prop- erty, ad their Martitct ideals have the propertl of zeros (and cousequetly are the (b) All P define the saine chamcteristie line field L = L(Pt). (e) Ail Pt satisf9 condition (A). The there eists a .family ( qf diffeomorphimns of ./a+ sendin 9 P fo Po. Remark. Recall that the extension prol)ery of S holds autonmticallv in the C caegory. % conjecture that in the C category the extension property in Theorem 1.3 also can be omitted. Our proofs show that this is so if le familv Pt h a generator that is polynonfial in t. Theorem 1.4. Let Po = (o) ad PI : () be Pfaff equations on M +, h 1. u,hieh bave the saine Martinet hypersurface S = S(Po) = ,ç(P) and the saine characteristic lie fleld L : L(Po) = L(P). Assune that the Mainet ideal of Po bas the property of zeros and Po satisfies condition (A). Then there eists a diffeomorphism sending P fo Po, provided that ' is su«iently close to 'o in the C topology. Remarh (Closeness of to 0)- As in the even-dimensional ce, often one can present a mmfl)er r < . (depending on P0) such that the closeness in the C topology in Theoreln 1.4 ca. t)e replaced by closeness in the C topology. Sec Theorem 0.2 and Theorem B.2 in Appendix B. The contents of the rucher .'ections. In Section 2 we explain whv the condition (A) is natural and give examples showing that it. cammt be veakened. The conse- quences of condit.ion (A) are explained in Section 3 and Appendix A: the condition (A) implies certain global division properties of a characteristic vector field. Sec- tion 4 contains auxiliarv algebraic statements, which also will be used throughout the proofs. Using the division properties and these algebraic staements, ve prove Theorems 1.1 and 1.3 in Sect.ions 5 and 7, respectively. The proofs of these the- orems are based on the honmtopy method, according to which it suffices to prove local" one should use Cartan's Theorem A in [C], which says that any local section of a coherent analytic module belongs to the module generated bv global sections. DISTRIBUTIONS OF CORANK 1 2865 the solvabilitv of the equation (HE) Lz, wt q- htcot q- d7 = 0 with respect t.o a family of vector tiens Zt and a family of fimctions ht (here Lzw denotes the Lie derivative of w along Z). Tllen the familv t of diffeomorphisms obtained by integrating the familv of vector fields Zt, dt - Zt(Ot), o = id, dt transfonns the Pfaff equations (wt) into (w0): êt w = tWo, where 't = exp(- ]ô t.ds) and t = ht o et- In what follows thc equation (HE) will be called the homotopg equatio or bomological equation. Theorenls 1.2 and 1.4 are proved in Sections 6 and 8 I)y reduçtion to Theorems 1.1 and 1.3. In these sections we show that if Pfaff equations Po and P satisfv the assumptions of Theorem 1.2 or 1.4. then tlwre exist generators w0 of P0 and of P1 suçh that the path of Pfaff equations Pi generated bv wt = w0 + t(w - w0) satisfies Ihe assmnptions of Theorcm 1.1 or Theorem 1.3. In Al)Iwndix B we present certain topological properties of linear operators re- lat.ed to the 5Im't.inet ideal and /he characteristic ideal. W also show a vav of transition from the assmnption of C-closeness of w to 0 in Theorems 1.2 and 1.4 to the C-closeness with a certain r < ,. In the simplest cases this wav leads to Theorems 0.1 and 0.2 in the lntroduçtion. _9. NECESSITY OF CONDITION (A) In this section we explain whv the condition (A) dv(P ) >_ 3 is natural, and we give examples showing that this condition cmmot be weakened: if depth dv(P ) = 2. then out theorems are hot truc anvmore. Fix a point p E AI a and denote by J the space of/-jets of 1-forlns at p. The condition that p is a singular point of the characteristic line field L, i.e. p G Sing(L), is the condition (u3 /k ((]oE)k--1)(p) = O, if n = 2k, and (,d A (dw)k)(p) = 0. (,/ (dc6) k-1 A dH)(V) -=- O. ifn = 2k+ 1. It involves the /-jet at p of a generator, of P, where i = 1 if z is even and i = 2 if n is odd. This condition distinguishes a certain subset of J--tlle space of/-jets of w at p. It is hot difficult to see that for any parity of n this subset is a stratified submanifold of codimension 3 (see [Mari, [Zh2], or [JZh2] for lnore details). Consequently, for generic , the set Siz9(L ) is either empty or a submmlifold of AI r of codimension 3. In the real analytic category (and conjecturally, in the smooth category too) the set of 1-forlns , violating (A) bas infinite codimension in the space of all 1-forms on AIr: see [JZh2], Proposition 3.4 and Theorem A2 (Appendix 2). The following examples show that the conditioll (A) cannot be replaced by the condition dp(P) OE 2. In these examples dira M = 4 and diln M = 5. They can be 2866 B. JAKUBCZYK AND M. ZHITOMIRSKII extended to higher dimensions. We have not round an example in the 3-dimensional case, but we believe that such an example exists. Example 1. Consider the family of Pfaff equations on the 4-torus T 4 generated by 1-forlns #t = dO + (sin 03 sin 04 + t)d02. The characteristic vector field Xt is the sanie for ail t: 0 0 Xt = X0 = cos 04 sin 0 3 33 COS 0 3 sin 04 004" The set S of singular points of X0 is the union of 8 disjoint 2-dimensional tori (4 of thcm are described by the equations 0a, 0a E {7r/2, -7r/2}, and the other 4 by the equations 0a, 0a E {0,-r}). The codimension of S is 2; therefore depth Ip = 2. The restriction of (t) to any of these 2-dilnensiolml tori is a Pfaff equation generated by a 1-form ct = dol + ( + t)d02, where 5 {0, 4-1} depending on the torus. This Pfaff equation eau be identified with the vector field Vt : (5 + t)O/O01 - 0/002 defined up to nmltiplication by a nonvanishing function. It follows that the phase portrait of I/ on the tortis is invariantly related to (at). It is well known that the equivalence of the phase portraits of 1 and t with a fixed 5 implies ri = t2 provided that t2 is close to tl; see [ArI1] (the parameter t corresponds to the rotation number). Therefore the parameter t of the family Pt is a modulus (a parameter varying continuously and distinguishing nonequivalent Pfaff equations). Example 2. Consider the family of Pfaff equations on the 5-torus T(O, 02, ¢1, ¢2, generated by 1-forms t = (A(0, 02) + Bt(01.02, ¢2)) d¢ + C(01.02)d¢2 + des, where A(01,02) = 3(sin 01 + sin 02), Bi(01,02) = t sin¢2 (1 - cos(0 - 02)), C(O. 02) = cos 01 + cos 02. A simple calculation gives that t A (dt) 2 = sin(01 - 02)-Qt" fL where f is a volume form and Qt is a family of nonvanishing functions on T 5, if t [-1.1]. Therefore the Martinet ideal is the smne for ail t; it is generated by the function sin(01 - 02). The Martinet hypersurface consists of two disjoint 4-tori: s= îu', î = {o2 =ol}, ' = {o2 =ol + }. Since the function Bi vanishes on the torus T14, the restriction of (ut) to T14 does hot depend on t. The restriction of (aJ) to the torus T_(¢1, ¢2, ¢3.0) depends on t: it is the Pfaff equation (ct), where (ct) = 2t sin ¢2d¢ + d¢3. The characteristic vector field of (t) restricted to T is the characteristic vector field of (ct). It is 2t cos ¢20/001. Assume that t #- 0. Then the characteristic line field does not depend on t. The set of its singular points is the union of two disjoint 3-tori T:ï:, given by the equations ¢2 = 4-7r/2. The restriction of (at) to T, 3 (or. the saine, the restriction of (ct) to T:ï:) is the Pfaff equation of the form (/3t),/3t = 2t5dçbl + d¢3, {-1, 1}. Consider the vector field Vt : -0/0¢1 + 2t0/0¢3 on the 2-torus T 2 = T2(¢,¢3). It is easy to sec that the Pfaff equations (,3) and (,3t2) on the DISTRIBUTIONS OF CORANK 1 2867 3-torus are equivaleut if and only if the phase portraits of t 1 and r 2 on the 2- torus are equivalent. As in the previous example, this is so if and only if ri = t2 provided that tu is close to tl. Therefore the paralneter t of the family (wt) of Pfaff equations is a modulus, although these Pfaff equations have the saine Martinet ideal (satis[ving the property of zeros) and the saine characteristic line field. The reason for that is the violation of the assulnption (A)- the dcpth of the characteristic idem is equal to 2 instead of 3. 3. CONDITION (A) AND DIVISION PROPERTIES In this section ve explain implications of condition (A) which will be essential in fm'ther proofs. The main implicatious are the following global division properties of a characteristic vector field X. As belote, we work in the C and C categories. Proposition 3.1.a. If a Pfaff equation P = (w) satisfies condition (A). then any characteristic vector field X of P bas the following division properties. (i) If n is even. then for any vector field " and any r-form u on ]I with r = n-1 or r = - 2 the equality X]u = 0 implies u = for an (r + 1)-form p on M', and the eqmlity X A t = 0 implies I = fX for a function f on M n. (ii) If n is odd. then for any vector field }" on 1I ' and any ( - 1)-form u on M' the equality xJu=o mod(H) imvlies u=xJg mod (H), for an n-form # on M , and the equality XAI=0 lnod (H) implies Y=fXmod (H) for a function f on M'. Here (H) is the Martinet ideal of P, and we assume that ( H) bas the property of zeros. This proposition is a corollary of a general theorem in [DJ] on division properties of the interior product with a section X of a vector bundle (see Appendix A for the proof). We also need a division property with parameters. In the next and all further st.atements in this section a 1-parameter falnily of fimctions, differential forms or vector fields on M is assumed to be re9ular in t, i.e., depending on t mmlytically (in the C category) or smoothly (in the C category). Proposition 3.1.b. Proposition 3.1.a holds with the forms and function f replaced by familles ut, #t, t't, ff, t [0, 1], provided that in the odd-dimensional case either the set S = {H = 0} bas the eztension property (sec Section 1) or the familles ut and t't depend on t polynomially. This proposition is also proved in Appendix A, using the already lnentioned general theorem on division properties. Remark. Proposition 3.1 also holds for germs at a fixed point. Another implication of condition (A) concerns the structure of the set Sing(L) of singular points of the characteristic foliation L: it cannot be too degenerate. 2868 B. .]AKUBCZYK AND M ZHITOMIRSKII Proposition 3.2. If a Pfaff equation (w) satisfles coditio (A) and in the odd- dimensioal case the Martinet ideal of (w) bas the propert9 of zeros, then any characteristic vector fietd X of(w), the Martinet hypersurface S and the set Sing( L ) of singular points o.f the characteristic foliation bave the followin 9 properties. (i) If n, = 2k, then the set M \ Si9(L) of points where X does hot vanish (i.e. () is quasi-cottact) is dense in M'. (il) If = 2h + 1. then the set iII \ S (i.e., the set of points af which is contact) is demse in M'. Equivalently, any 9enerator H of the Martinet ideal is ot a zero divisor. (iii) If = 2k + 1. the the set S \ Si9(L) is dense in S. Pro@ Statelnent (i) follows froln thc observation that if a is liOt quasi-contact at ails" point of ali open set, then any charactelistic vector field X vanishes on this set. (vanishes on a connected COlnponent of M', in the analvtic category). Consequently, givell a point p in this set. the charactcristic ideal Ip at p generated by the çoetïficielits of X is trivial and contains no llOn-zero-divisor. This COlitradicts assmnl)tion (A). Statolnent (ii) is a simt)le inq)lication of the property of zeros of the Martinet ideal: see Definition 1.5 raid the exami)les following it. To prove (iii), assulne that there exists a neighbourhood U in 3I" of a point p G S such tliat a characterislic vector field X vanishes at any point of the set U ç/S. Bv the property of zeros of tlie Martillet ideal we obtain that Xp = 0 nlod(Hp), where thc subscript indicates the gerln at p. This contradicts assulnption (A) at the point p. The proof is COlnplete. [] Propositions 3.1 and 3.'2- inlply the possibility of choosing t he Saille characteristic vector field for all Pfaff equations with the saine characteristic foliation. Proposition 3.3. Let Pt = (--'t), t [0.1], be a famil9 of Pfaff equations on Iii ' satisfyin 9 the assumptions of Theorem 1.1, if » = 2h, or of Theorem 1.3. if = 2k + 1. Then for any family Xt of characteristic vector fields of (wt) we bave Xt = RXo, if = 21,', or Xt = R Xo nlod (H), if = 2k + 1, where Rt, t [0.1], is a family of positive-valued fumtions and Xo i.s Xt with t = O. Pro@ Let n = 2],'. Tlie equality L(at) = L(a0) inlplies that (X, A Xo)(p) = 0 for ail p G M'. By Proposition 3.1.b we obtain tliat X = RXo, where R is a family of functions. Proposition 3.1.b also ilnplies that for any fixed t we liave X0 = QX, where Qt is a fimction on _M . This leads to the relation (1 - t?Qt)Xo = O. Bv Proposition 3.2. X0 does not vanish on a dense subset of M'; thus RtQ = 1. This irai)lies that Rt is a family of nonvanishing fimctions. This falnily is positive value& since for any p G/il the fimction t?t(p) is contiliuous in t and Ro(p) = 1. In the case of r = 2k + 1 the equality L(wt) = L(wo) gives (Xt A X0)(p) = 0 for all p G 5'. From the property of zeros of the Martinet ideal v«e deduce that Xt A X0 = 0mod(H). Using Proposition 3.1.b, we see that Xt = t?Xomod(H), fol" a fainily of fimctions Rt. Similarly, we have X0 = QtXt mod(H) fol" any fixed t, where the Q are filnctions. Therefore, (1 - t?Qt)Xo = 0inod(H). Bv Proposition 3.2, I?Qt = 1 on S, and so/7 is nolvanishing on 5'. From the fact that Rt(p) is continuous in t and ri'oin t?o(p) = 1. we deduce that Re is positive vahled on S, DISTRIBUTIONS OF CORANK 1 2869 for anv t E [0, 1]. Finally, adding to Rt the flnction ('l-I 2 with a sufficiently large constant C, we obtain Rt positive valued on/I. [] Condition (A) implies one lnore division property that xve need in our proofs. Its proof is postponed to Section 4. Proposition 3.4. Let Pt = (wt). t ff [0.1], be o fo,nily of Pfaff eqaatio, on 3I 2k satisfying the ass'umt)lio'ns of Theorem 1. l, and let fit be a family of 1-forms such that wt A Pt = 0. Then dt = h.twt .for some family ht of fu,wtions. Note that this sta.tement is trivial if wt is a falnilv of lonvanishing 1-forlns. but we do hot assume this in our theorems. 4. A('XILIARY ALGEBRA1C LEMMAS To prove the solvability of the homot(q)y equation in the pro(frs of TheorellS 1.1 and 1.3. we will also usc the following siml)lç algei)raic facts. Rccall that a 1-form e on 3I ', n _> 3. is called contact (quasi-contact) at p Ç M' if n = 2/,: + 1 (resl)ectively, n = 2k) and ((t A (da))(p) # 0 (respectively, (c A (d(@-l)(p) ¢ 0). Lemma 4.1. Let c aztd A be 1-forms on M 2. ff c is quasi-contact at p and (,x / o/ (do) -2) () = ,I. te ( / et)() = O. Lelnlna 4.2. Let o be a l-form on/i2k+1. If ( is a contact al p and A is a 1-form such that (A A et A (d(Q k-l) (p) = 0 and (A A (da) k) (p) = O. then A(p) = O. The facts stated in these lelmlmS are invariant with respect to multiplication of a by a nonvmfishilg function, i.e., they are properties of the Pfaff equation (c). These properties Call 1)e easily checked in the Dari)oux coordinates in which the Pfaff equation takes the forlu (dz + .rldyl +"- + .r,.dy), where r = and r=kifn =2/,:+1. Lemma 4.3. Let c be a l-.form on ri/2k+l that is hot contact at p, but c(p) ¢ O. If A is a 1-form such that (A A a A (do)-)(p) = O. then (A A (da))(p) = O. Pro@ We take a nonzero vector v Ç Tpdl 2k+l such that t, Jct = vida = 0. (The existence of such a vector follows ri'oto the assumption that o is not contact at p.) Then the relation assunled iii the lelmna implies that the form (viA) c A (dc@ -1 valfishes at p. It follows that if (o A (da)-)(p) ¢ I). then (vJA)(p) = 0. md consequently ( A (da)k)(p) = 0. On the other hand. if (a A (da)t'-1)(p) = 0. then the assumption c(p) ¢ 0 implies that (da)(p) = 0, and then again (AA(da)k)(p) = 0. [] Lemma 4.4. /f ft is a volume form on Al ', A is a 1-form, 7 is an (n - 2)-form and X is a vector field defined by the relation X J f = A A 7, then X ] A = O. Pro@ To prove this statelnent, note that the definition of X ilnplies X J (AAT) = 0, and consequently (X ]A) -7-t- (X ]7) AA =0. It follows that (X ] )" (AAT)=0. Since X valfishes exactly at points at which the form A A7 vanishes, we obtain that xii=o. [] Finally, we need the following general properties of a characteristic vector field. 2870 B. JAKUBCZYK AND M. ZHITOMIRSKII Lemma 4.5. If X zs a characteristic vector field of a Pfaff equatzon (w) on 1I 2k, tl en X]w = 0 and (X]d,z) Aw = 0. Pro@ The first relation follows ri'oto the definition of X and Lemma 4.4. The definition of X implies that X] (w A (dw) '-) = 0, which, together with X]' = 0, gives (XJdw) A w A (dw) k- = 0 if k > 1. Now the second relation in Lemma 4.5 follows froln Lemma 4.1 at points where w is quasi-contact. At all other points the field X vanishes, and thcre is nothiug to prove. Lemlna 4.6. [f X is a characteristic vector field of a Pfaff eqatio (w) on M 2+1 wlose Martinet ideal bas the propertg of zeros, and H is a generator of this ideal. then XJw=0 mod (H), XJdH=O mod (H), (XJdw) Aw=0 mod (H). Pro@ Due to the I)roperty of zeros of (H), it suffices to prove the three relations at any point p of the Martinet hypersurface S such that X(p) O. The relation (XJdH)(p) = 0 follows immediately from the definition of X. To see the other two relations, note that S is regular in a neighbourhood of a point p such that X(p) O. The definition of the characteristic vector fiel(1 X in the case n = 2k+ 1 implies that the vector field X]s on S is, in a neighbom'hood of such a point p, a characteristic vector field of the Pfaff equation (w]s) on S (which is qui-contact at p). Thus the remaining two relations follow from Lemma 4.5. The proof is complete. Proof of Proposition 3.4. Let Xt be the characteristic vect.or field of (wt) defined bv XtJ = wt A (dwt) -. Since Xt (and so wt) does not vanish on a dense sub- set of al , the condition t A/t = 0 and Lemma 4.5 imply that Xt]flt = 0 and Xt] (fit A (dwt) k-I) = 0. From Proposition 3.3 we have the equality Xt = RtXo, with Rt nonvanishing; thus X0J (fit A (dwt) k-l) = 0. Therefore the division prop- erty in Proposition 3.1.b implies the following relation: fit A (dt) k- = No ] Pt = (gt/Rt)Xt ] , where is a volume form, gt is a family of flmctions and t = gt. Taking ht = gt/R, we tan rewrite this relation iii the form (t - brut) A (dt) - = O. Let us show that this relation implies fit- tt = O. know that (fit- htwt)Awt = 0, since fit A wt = 0. Fix t and a point p at which wt is qui-contact. At this point wt does hot vanish: therefore (fit - btwt)(p) = rwt(p), with the scalar r depending on t and p. Then the displayed relation implies that r(wt A (dwt)k-)(p) = 0 and consequently r = 0. So, (fit - htwt)(p) = 0 if p is a point at which wt is quasi- contact. By Proposition 3.2. (i) the set of such points is dense, and so fit = htt at any point of the manifold. The proof is complete. Now we are ready to prove the solvability of the homotopy equation (HE) and out main theorems. 5. PROOF OF THEOREM 1.1 Solvability of the homotopy equation (HE) in Section 1 is equivalent to solvability of ( dwt) )k-2 (5.1) Lz, a.'t ÷ -- Awt A (dw't =0, DISTIRIBUTIONS OF COIRANK 1 2871 with respect to a fmnily Zt of vector fields. NaInely, equation (5.1) is obtained from the homotopy equaltion by extlernal multiplication by wt A (dt) -2. Cnversely, if (5.1) is solvable then, using the fact lhal the set of quasi-contacl points of (wt) is dense in BI" (Proposition 3.2.(i)), we get from (5.1) by Lelnma 4.1 that (Lztwt + (dwt/dl))Awt = 0. Therefore, by Proposition 3.4 we get Lz, wt +(dt/dt)+htt = 0. for a fmnily of fllnctions ht, which is the holnotopy equat.ion (HE). A solution Z, of equation (5.1) will be constructed within the set of fmnilies Zt satisfying (.2) z, ] , = 0. Condition (5.2) ilnplies that Lz, wt = Zt ] dwt, and the equation (5.1) tan be rewritten in the forln dt )k- (5.3) Z](wtA(dt)-l)+('-l)AwtA(d, =0. In order to solve equation (5.3} we fix a vohlnm form tl and define a familv Xt of characterist.ic vector fields of (wt) by the relaion Xt ] Q = wt A (dwt) k-l. Lemlna 4.5 and Proposition 3.3 imply the relations XoJwt = 0, XoJ(dwt/dt) = 0 and (Xo]dwt) A wt = 0. Thus X0]ut = 0. where oet = Awt A (dt) - Therefore, by the division property in Proposition 3.1.b, we ham with solne Nlnily fit of (n - 1)-forms of the saine regularity with respect to t as in wt. Using Proposition a.a again, we obtain for some, regular in t, family of ( -- l)-forms pe. This relation allows to rewrite equation (5.3) in the form &](X,] )+(-I)X,],=0. The latter equation bas a solution Z« defined by the relation It now renmins to check that the construçted solution Z satises relation (5.2). The equality (5.2) is equivalent to the relation A 0. From (5.5) and the definition of oe we bave (X ] ) A 0. By Lemma 4.5. XJwt 0; therefore X ] ( A ) 0. Thus the n-form A « vanishes at any point at which X« does not vanish. By Proposition 3.2, (i) the set of such points is evervwhere dense; therefore p« A 0 and (5.2) holds. This completes the proof of Theorem l.l. 6. PROOF OF THEOREM 1.2 We will use the following proposition (its proof is postponed to the end of this section). Proposition 6.1. Assume that Po = (w0) satisfles condition (A) and (6.1) w/ (d) k-1 = w0/ (da0) k-l. 2872 B. JAKUBCZYK AND M ZHITONIIIRSKII Tbe for the patb = (1 - t)wo + t we bave (6.2) cor A (dct) k-1 = A t 0 A (d0) k-l, where Af is a family of functions, polynomial i t. Proof of Theorem 1.2. The equality L(0) = L() implies X0 A X = 0. where Xo and X are characteristic vector fidds of P0 and P1- Condition (A) satisfied for (0) enables us to use the second division property in Proposition 3.1.a, (i) to dedu«e that X = RXo and. equivalently, (6.3) A (dWl) k-1 = R0 A (dw0) k-l, where R is a Slnooth or analvtic flmction. In fact, R is positive value& which wil] follow ffoto the closeness of wa to w0- Therefore. assunfing R > 0, we choose the gellerator and we have (6.4) Let (6.5) £1 / (d&l) k-1 = '0 / (dw'o) k-l- t = (1 - t)co0 +/I- To prove Theolen 1.2 it is sufficient to show that the family of Pfa.ff equations satisfies the assumptiols of Theorem 1.1. The equality (6.4) allows us to use Proposition 6.1 to conclude that the relation (6.2) holds for the path (6.5). It is «lear that (6.2) implies that the familv (wt) satisfies the assmnptions (a) and (b) of Theorem 1.1 provided that the fun«tions At in (6.2), t G [0, 1], vanish at no point of 3I . This will follow froln the assump- tion on the C-closeness of wt t.o 0 and Theoreln B1 in Appendix B. Define a characteristic vector field Xt of (wt) bv the relation XtJQ = t A (&t) -1. where Q is a vohune forln, By (6.2) we bave Xt = AtX0. The C-closeness of '1 O '0 implies the C-closeness of Xt, t Ç [0, 1], to X0. Bv Theol'eln B1 the C-closelmSS of Xt to X0 in the equality Xt = AtXo ilnplies that the function At, t [0.1], is C-«lose to 1. ÇOlsequently, Af valfishes at no point of the lnanifold. The proof of Theorem 1.2 is COml)lete. Proof of Proposition 6.1. Using (6.1), we may assume that the chara«teristi« vector fields Xo and X of (#0) and (#), respectively, are equal. shall prove that (6.6) X0 J (w, A (dwt) k-l) =0. Having (6.6), we can use assumpt.iox (A) and the division property in Proposition 3.1.b (wit h polynolnial dependence in t), which gives wt A (dwt) k- = XoJ Pt, where pi is a volume form. Let 'o A (d#o) = Xo]. = A. Then we get (6.2). To pi'ove (6.6), we note that bv Lelmna 4.5 we have (.7) XoJ,o = xOJ«l = 0; therefore XoJwt = 0. It follows that in order to prove (6.6) it suffices to prove the equality It is enough to prove the equality (6.8) at any point p such that Xo(p) ¢ O. At such a point o(P) 0 and, since Xo = X1, (p) O. From Lelnlna 4.5 we have DISTIRIBUTIONS OF COIRANK 1 2873 (XoJdwi) A wi = 0, i = 0, 1. Thus, there are flulctions h0 and hl, defined in a lieighboul'hoo«l of p, sUC|l that iii this imighbourhood ve bave (6.9) XoJdwo = howo. XoJdw = hll. XX will prove below that (6.10) h t = h. Thell frolll (6.9) we get XoJdwt = hwt, where h = h0 = bi, and so (6.8) holds. We will show that (6.10) follows ffoto (6.1}. take thc Lie derivative of both parts in (6.1) along the veçtor field X0. Using thc forlnula Lxq = d(XJ q) + XJdq for the Lie derivative, we obtain = 0 + k(X0Jdw0) A (da0) -1 = t'haa0 A (dwo) -1, and siinilally L.,- (1 (dl) -') = t,,l (1) *-1. Conq)al'ing these equalities and using (6.1) again, we gel the required relation (6.10} (since w0 A (dwo) t- = w A (dw) t-I does llOt vanish on a dense subsçt of 3I). Proposition 6.1 is proved. 7. PIROOF OF T.HEOREM 1.3 Since the lklartiiml hypersurfaces of Pt = (wt) are the samc for all t. the lklartinet ideals are the Saille bv t.]le I)roI)erty of zeros. Thus we Call fiX a generator H of these ideals. The following two propositions will hold ulder the assumptions of Theorenl 1.3. In the propositions ail fanlilies are regular with respect to t (slnooth in the C category and almlytic iii the (7" category). Proposition 7.1. There ezists a familg of vector fields l't satisfging the relation ( (7.1) Ly, w, + dt J Awt A (dw,) k-1 =0 mod (H). Proposition 7.2. Let pt be a familg of 1-forrns such that (7.2) tzt A Wt A (dcdt) k-1 = 0 niod (H). Then the equation (7.3) Lztwt bas a solution ( Zt, ht ) . The solvability of the honiotopy equatioli (HE) in Section 1 is a direct corollary of these propositious. Nalnely, we take ttt = --Lytcd t -- dwt/dt, and then the pair (2t, ht), with 2t = Zt + t, solves the holnotopy equation (HE) Proof of Proposition 7.1. We fix a vohlme form t and define a falnily Xt of char- acteristic vector fields of (wt) by the relation XtJf = wt A (dwt) k-1 A dH. Frolll Proposition 3.3 we have X t = tt.\" 0 lllod (H), where/t is a familv of nonvmiishing fllllctiollS, regular in t (of the saille regularity in t as in the falnitv wt). Bv Lenima 4.6 we bave XtJwt = 0 mod (H) and {XtJdwt) A wt = 0111od (H). V'P Illay repla.ce X with Xo iii these equalities, hl 2874 B JAKUBCZYK AND M. ZHITOMIRSKII particular, we get X0]wt = 01nod (/-/), which ilnplies Xo](dwt/dt) = [Imod (H). Taking ail these equalities into account, we see that (7.4) X0] Awt A(dwt) -1 = 0 mod (H). This equality and Proposition 3.1.b inlply that dwt d A wt A (dwt) k-1 = Xo J (f,) mod (H), where ft is a falnily of fimctions, regular in t. Replacing X0 with RÇ1,Yt and using the definition of Xt, we see that we can rewrite this relation in the forln A wt A (dwt) k-1 dt where gt = ft/Rt. This allows us to rewrite equation (7.1) in the form (7.5) (L,wt - gtdH) A wt A (dwt) -1 = 0 mod (H). It is clear that (7.5) hohls if satisfies thc relations (7.6) l) J dwt = O, kt J wt = gtH. since in this case Lwt = d(})J w) = d(gtH) = gtdH mod (H). Since (H) is the Maltinet idem of (wt), we have (7.7) for a falnily St of nonvanishing functions which has the sanie regularity in t in wt (this follows froln the regularity of the leh-hmld side and the fact that division by H is a continuous linear operator in the space of Slnooth fnllctions, see Theorem B1 in Appendix B). Let us show that (7.6) holds for the family }) defined by (7.s) ] = (d) . In fact, applying }] to (7.8), we get gt(}Jdt)A(dwt) - = 0. This relation ilnpli J dwt = 0 (at points where g,(p) = 0 we have }(p) = 0, and at other points we can use Lelnnla 4.2 with A = (]dt) and the fact that contact points are dense). We have shown the first equality in (7.6). hl order to prove the second one we apply })] to (7.7) and, using (7.8). we obtain that (}] wt- gtH)" (dwt) = 0. This implies that ) ] wt - gtH = 0 at points where the form (dt) k does not vanish, in particular, at points where wt is contact. By Proposition 3.2, (ii) the set of such points is everywhere dense; therefore }]wt -gtH = 0 everywhere, and so (7.6) holds. Proposition 7.1 is proved. Proof of Proposition 7.2. Bv Lemma 4.2, (i) and the fact that the set of contact points is dense in BI (Proposition 3.2. (ii)), the equation (7.3) reduces to the following tvo equations: (7.9) (z,) , (d,) - = ,, ' (d.,) «, (7.10) (Lz, w, + h,,) / (d,,) k = ,, / (d,,) (with unknown Zt and ht), obtained from (7.3) by external multiplication by the forlns wt/ (dw,) -1 and (dwt) , respectively. DISTRIBUTIONS OF CORANK 1 2875 To solve equation (7.9) we use assmnption (7.2). By this assumption (7.11) Pt A cor A (dwt) k- = Hut for some familv u of 2k-forms, regular in t })y Theorem B1 in Appendix B. This permits us to find an explicit solution Z of (7.9). Namely, since H is a generator of the Martinet idem of (w), we bave relation (7.7), i.e., w A (dw) = H&[, where & is a fmnily of nonvanishing flmctions, regular in t. Let ris show that the family of vector fields Z defined by the relation k is a solution o[ thc equation (7.9). e]at.ion (7.1]) and the fact that is hot a zem divisor imply that ut A wt = 0. This and (7.12) imply that z, ] , = (. Consequently, Lz, wt = Zt ] dwt and (r.la) (Lz, co,)co,(d,) -' = (Z, J d,),(a,) -1 = Z, J (#, (d,/). k Now (7.9) follows from equalities (7.7) and (7.11)-(7.13). To prove Proposition 7.2 it renmins to solve equation (7.10) with respect to Since Zt]wt = 0. then (Lz,t) A (dt) = (k + 1)-Zt](dt) TM = 0, and the equation (7.10) takes the form htwt (d,) = vt (d#t) . Due to relation (7.7), to prove that this equation has a solution ht it suces to prove that #t A (dwt) = HCt, where Ci is a family of fimctions, regular in t. shM1 first prove that (r.14) (,(d,) )(v)=0. f VeS- This follows from relation (7.2). Namely, since t is hot contact at a point p S. thus (7.14) follows from (7.2) bv Lenmm 4.3. provided that wt(p) ¢ 0. Since the set of points of S at which wt vanishes is a subset of the set Sig(L), the set of points p S where wt(P) ¢ 0 is dense in S by Proposition 3.2. (iii). Therefore (7.14) holds at ail points p S. By the property of zeros of the ideal (H) we obtain t A (&t) = HCtiL Since t and t are regular in t, we deduce from Theorem B1 in Appendix B that Ct is regular in t. Proposition 7.2 is proved. We have completed the proof of Theorem 1.3. Note that the extension property of S was used only when referring to Proposition 3.1.b., and therefore it is hot needed if cor is polynomial in t (cf. the remark after Theorem 1.3). 8. PROOF OF THEOREM 1.4 Since the Martinet hypersurfaces for Po = (oz0) and P = (COl) are the same and the Martinet ideals have the property of zeros, they are equal and we tan choose a common generator H which will be used throughout the proofs. The following proposition holds under the assumptions of Theorem 1.4 and will enable us to reduce the problem to Theorem 1.3. 2876 B JAKUBCZYK ANDM ZHITOMIRSKII Proposition 8.1. Assume that OE1 / (doE1) k-1 / dH = wo A (dwo) k-1 A dH Ttem for the potl t = (1 - t)wo + hZl we Iave Wt A (dwt) k-1 A dH = Bt wo A (dwo) k-1 A dH (8.1) (8.2) mod (H) mod (H), SO 1 A(doel) k-1 A dH = co0 A (dw0) k-1 A dH mod (H). cor = (1 -- t ) wO q-tdOl. To prove Theoreln 1.4 it is enough to show that the family of Pfaff equations (wt) satisfies the assuml)tions of Theorem 1.3. Note that we do hot need the extension property of ,5'. since the familv cor in (8.6) is polynomial (in fact. affine) in t, and in this case Theorem 1.3 was I)roved without using this assmnption. The equality (8.5) enables us to use Proposition 8.1. h is clear that relations (8.1) and (8.2) imply that the falnily (wt) satisfies the assmnptions (a), (b) and (c) of Theorem 1.3 provided that the fimctions Bt do hot vanish on S (then the characteristic line field does hot change) and Ct do hot vanish on 3I ' (then the Martinet ideal does hot change), for t [0, 1]. The fact that Bt and Ct do hot vanish follows from tlle 0 and Theorem B1 in Appendix B. Since CO 1 is C-close field Xt is C close to X0. The relation X = /?X0 mod the inverse to the operator Lx.H in Theorem B1 imply that /) that is C-close to 1 and equal to at anv point of the S. Bv the property of zeros of the Martinet ideM, /) = replace bv / in (8.3) and in the definition (8.4) of &l- CC-close to CO0- Define a falnily of characteristic vector fields Xt bv the relation XtJçt -= COr A (dCOt) k-1 / dH. Then Xt,t [0, 1], is Coe-close to X0. The equality (8.1) is equivalent to Xt = BtXo mod (H). We again use continuitv of the inverse to the operator Lx.H in Theorem B1. By this theoreln there exists a fimction /t that is C-close to 1 and equal to Bt at any point of S. Therefore Bt > 0 at any point of S. (8.4) and we gct (.5) Let (S.) COr A (dcot) k --- Ct CO0 A (dCO0) k, wher'e Bt ad Ct are fizmilies of flnctions, polynomial in t. Proof of Thcorem 1.4. Let X0 and .k' be characteristic vector fields of P0 and P defined via the saine volume tortu and thc saine generator H of the Martinet ideal. Since L(wl) = L(wo), then X A X0 = 0mod (H). Froln condition (A) and thc division property in Proposition 3.1.a we obtain X = HX mod (H) or, equivalent ly, (8.3) 1 (dwl) k-1 dH = RWo A (dw0) k-1 A dH mod (H), where is a smooth or analvtic flmction. will later show. using closeness of w to w0. that is positive vahmd. Thcrc%re, we can change the generator of Pt for 1 = R/w, DIST1RIBUTIONS OF CO1RANK 1 2877 To prove that Ct vanishes at no points of àl n, we also lise the C°°-closeness of cet. t E [0, 1], to w0 showll above. Let wt A (dwt) k = QtHf, where is a volume form. Relation (8.2) iml)lies that Qt = CtQo. The C°-closeness of cet to co0 implies the C-closeness of the fimction HQt to HQo. By continuity of the inverse to the operator f ---, fil (Theorem BI) we get the C-closeness of Qt to Qo. Since Q0 is a nonvanishing flmction, tlwn Ct is Coe-close to 1. The proof of Theorem 1.4 is complete. [] Proof of Propositio 8.1. It is enough to prove the equalities -\'o J (cetA(dce) k-1AdH)-----Il mod (/4), Namely, eqnality (817) and conditi»n (A) allow us to use the divisim properties in Proposition 3.1.b to conchlde that cor A (dcot)/'-1 A dH = Bt.\']t2mod (Il), and so the validitv of relation (8. l), where Bi is a famih of fimctions, polynomial in t. Equality (8.8) ilnplies that cor A (dcot) vanishes at those points of S at which X0 does hot vanish. By Proposition 3.2, (iii) the set of snçh poiuts is dense in S: therefore cor A (dcet) vanishes at all points of S. By the property of zeros of the Martinet ideal we have cet A (dcet) = 0 mod (H), and consequently (8.2) holds for SOlne familv of fnnctions Ct. polynomial in t. In order to prove (8.7) and (8.8) we use the assmnption of Proposition 8.1 and choose characteristic vector ficlds X0 and XI of (co0) and (coi) eqnal modulo (H). By Lelmna 4.6 we have .\]w, = 0 mod (H). XoJdH = 0 mod (H). for i = 0, 1, and therefore X0Jwt = 0 mod (H). It follows that in order to prove (8.7) and (8.8) it suffices to prove the equality (819) (Xo]dcet) A cet : 0 mod (H). Due to the property of zeros of the Martinet ideal, it suffices to prove this equality for any point p Ç S such that Xo(p) 7 O. At snch points the 1-fonn dH does hot vanish and S is smooth. Since XI = -\-o mod (H), using Lemma 4.6 we obtain (Xo]dceo) A o = (X0Jdce) A co I : } in a neighbom-hood of p in S I Since X0(p) ¢ 0. then oto(p) -¢ 0 and cel(P) ¢ 0, and therefore these relations imply the equalities (8l 10) XoJdce0 = hr.cco, XoJdvl --- ]licol, which hold iii a neighbourhood U of p in S. Here ]10 and ]il are functions defined in this neighbourhood. We will show that ho = hl; then (8.10) implies that Xo]dcet = ]10cot, and (8.9) holds in the neighbourhood U. To prove that ho = ]il Oll []" C S, we restrict the relation assmned in Proposition 8.1 to the tangent bundle of . We obtain (coi A dcel)[u = (co0 A (dco0)k-)l and take the Lie derivative of this relation along the restriction Xols of X0 to S (recall that X0 is tangent to S). As iii the proof of Proposition 6.1, we obtain the required equality ]10 = bi. Proposition 8.1 is proved. [] 2878 B. JAKUBCZYK AND M. ZHITOMIRSKII APPENDIX N. DIVISION PROPERTIES In this Appendix we present a general theorem on division properties of the exterior (respectively, interior) product with a section X of a vector bundle. This theorem is proved in [DJ] and implies our Propositions 3.1.a and 3.1.b. Our results hold in the categories C s, where s = ex2or s = w. Let AI be a paracompact differential manifold. Consider a vector bundle E over 'I of tank m and denote by E* its dual bmdle. Let Af = Af(E) denote the rth exterior power of E, r = 0, 1 ..... m, with A0 =/I x and Ai = E. We denote , A(M: E) the linear space of sections of A,, (smooth or real analytic, depending on the category). Any section cv of E defines lhe linear operator of exterior multiplication bv c. which gives the complex (A.I) 0 --, A0(M) ---, A(M: E) --, ...... --, A,,(M: E), wflh the operator 0« = 0 : Av(M: E) --, Av+(M: E) defined by 0(')') := o A % Consider a section X of the dual bundle E*. This section defines the operator of the interior product with A', Ni : A,.(M: E) --, A_I(M:E). Given a local basis e .... , e of E, the operator of the interior product with X is defined on the elements of a local basis of A bv xJ(% ... e) = (-1)-' (x. %) «, ...% ... e, 3=1 where êi means absence of ei mld (-, .) denotes the dualitv product between E* and E. Clearly, (X])2 = 0; so the operator A'] defines the complex (A.2) 0--, A,(AI: oe)--, A,_a(AI: oe)--, ...- Al(AI:E)-* Ao(M: E). Let S be a closed subset of .àl. Denote by A(M. S: E) C A,.(M: E) the subspace of sections of Af(M: £) vanishing at ail points of S. and let A(S: E)= A,.(AI: E)/A,.(AI, S; E) denote the quotielil space. Any element of AI (ç; E) defines the unique operator 0 : Ap(S: E) --, Ap+a (S: E) (the quotient of the operator of exterior multiplication), which gives the complex (A.3) 0--, Ao(S)--, A,(S: E)--, ...... --, A,(S: E). Given a section X of £*, the operator XJ defines the following complex on the quotient spaces: (A.4) 0--- Ara(S; E) Am-I(S; E)---.--- A(S: E)-- A0(S: E). We define the invariant dp(X) = depth(Ip), where Ip is the idem of function germs at p G M generated by the coelïïcients a ..... a of X in a local basis of E* (cf. Definition 1.2 in Section 1). Sinfilarly, given a pair (H, X) of a flnction H and a section X of E* on M, we define dp(H. X) as the maximal length of a regular sequence of function germs that begins with the germ Hp of H at p and has further elements in In (cf. Definition 1.4). Analogously we define the invariants dp(a) and dp(H, DISTRIBUTIONS OF CORANK 1 2879 Statements (i) and (il) of the following theorem hold in the C and C cate- gories, for 0 _< q _< n- 1. Theorem A. (i) If « satisfles the condition dp(o) >_ q + 1 for all p E .I such that c(p) = O. thon the complex (.4.1) is exact up fo Aq(M:E). Similarlg. if dp(X) >_ q + 1 for all p ]il such that X(p) = O. then the complex (.4.2) is exact up to Am_q(M; E). (il) Let H be a function on ]il such that lhe ideal (H) bas the prop«rty of zeros. and let S = {H = f)}. If et is a section of E on M such that (H.() satisfies d(H. c) >_ q + 2 for all p S such that et(p) = O. thon the conplex (.4.3) is exact up to Aq(S; E). Similarl,q. if X is a section of E* on M and d(H, X) >_ q + 2 for all p S such that X(p) = O. then the cornplex (.4.4) is e.ract q fo A,_q(,ç; E). (iii) If the assumptions of (i) hold, then. in the C category, the cornplex (A. 1) splits up to Aq-l(]il;/) and assumptions of (il) hold and splits up fo Aq-l(S;/) and t/Le complex (.4.2) split. up to Am-q+] (]il;/). If the S bas the extension propertg, then the conpleï: (A.3) the complex (A.) splits up to Am-q+l(S; E). Here the correspoding spaces are equipped with the C topology and are considered as Fréchet spa«es (quotient FfCher spaces). Above, a complex 0 -- L, -- .-. -- Lrn-q+l -- Lm-q -- ..- defined by the operators Oi : Li -- Li-1 is called exact up to Lm-q if Im0i+l -= ker Oi for i = m, m-- 1 ..... m --q, and it splits up to Lin-q+1 if the L are linear topological spaces. hnOi are closed subspaces of Li-1 and each Oi : Li -- Li-1 has a continuous right inverse tçi defined on hn Oi, for all i = m, m - 1 ..... m - q + 1. The above theorem follows rioto The(»rems 2.1 and 2.2 in [DJ]. In the local case (of germs) statements (i) and (il) follow from a well-known algebraic result on exactness of the Koszul complex, cf. e.g. lE] or [JZh2], A1)pendix 1. Proof of Proposition 3.1.a. In the even-dimensional case the first implication fol- lows trivially from statement (i) in Theorem A if we take the bundle E equal to the cotangent bundle E = T'M, the dual E* = TM, and consider the complex (A.2). (A(M, E) is identified with the space of diffcrential r-forms on ]il.) The second implication follows analogously from the saine statement by taking E = TM and the complex (A.1). In the odd-dimensional case the first implication follows in a similar way from statement (il) in Theorem A concerning the complex (A.4). This is because the property of zeros of (H) allows us to identify the elements of A(S: E) with the equivalence classes of differential r-forms modulo (H) (cf. out convention on no- tation mod (H) presented after Definition 1.3). The second implication follows analogously from statement (il) in Theorem A concerning the complex (A.3). [] Proof of Proposition 3.1.b. The existence of pt and ft for any fixed t follows from Proposition 3.1.a. We have to show the regularity of these familles in t. We shall prove the regularity of ttt (the proof of regularity of ft is analogous). If ut depends on t polynomially, then Proposition 3.1 allows us to construct #t polynomial in t, and the regularity follows trivially. In the general case out argmnents are different for the categories C" and C . In t.he C" category we use the following fact: if a sequence al ..... a, of rem analytic function germs at p ]il is regular in the ring of analytic function germs at p, then it is regular when considered as a sequence in the ring of real analytic function germs of the variables (x, t) M × 11¢ at (p, to), for anv t0 6 [0, 1]. Using 2880 B. JAKUBCZYK AND M. ZHITOMIRSKII local coordinates, this fact can be easily proved for the case of formal power series using the definition of regular sequcnce in the ring of fornml power series of the variables x ..... xn, t. Then, using the fact that the ring of formal power series is faithfully fiat. over the ring of convergent series (see Malgrange [Mlg], Chapter 3), we see that it also holds for converging series and so for germs of analytic functions. Using the above fact we see that the assumption (A) holds over the manifold ,ÇI = M x I, where I is an open interval containing [0, 1] on which the analytic family vt is well defined by analytic extension. Thus we Call lise Theorem A over the manifold .). i.e., for the bundles E = T*M and E* = T3I pulled back to IÇI by the canonical projection M x I 21I. In the (' category, in the even-dimelsional case the smooth dependence of pt on t follows rioto statement (iii) in Theorem A. By this statement there exists a continuons right inverse operator K to the linear operator X] : A,.+(M: E) A,.(M: E), for r= n-1 and r = n-2, and we can define pt = Kut. Here E = T*M and A,.(M: E) = A(M). the space of diffcrential r-tbrms on M. In the Coe category, iii the odd-dilnensional case we also use statement (iii) of Theorem A and the extension property of S. Namely, fbr E = T*3I we define lt = A Kut]s, where Iç : A,-I(S; E) A(S: E) is the continuous right inverse p,'t,r to xJ : .,,(s; ) &,_(s; ). ,,d : .%(5'; ) .,,(M: ) is contimums linear operator of extension. APPENDIX B. CONTINUITY OF DIVISION Contilmity of division in the cases presented below is needed in the main proofs and will be proved separately. Let C(.I) and l'ect'(M) be the spaces of smooth functions and smooth vector fields on M. with the C topology. Let C(3I, S) and lect'(M..ç) be the subspaces of flmctions (vector fields) on 3I vanishing on the Martinet hypersurface S c M. The quotient Fréchet spaces C(.ç)=C(M)/C(M..ç), Vect(S:TM)= Vect(M)/Vect(M.S) can be idcntified with the space of smooth function on S and the space of blnooth sections of the tangent bundle T3I restricted to S. rcspectively. Given a Pfaff equation on 3I 2 and a characteritic vector ficld X, we consider the linear operator Lx : C(M) -+ lect(M). Lx(f) = fX. For a Pfaff equation on M t+. a characteristic vector field X and a generator H of the Martinet idem we consider the linear operators /: C(M) C(M) ;(I) = and where [ ] denotes the equivalence class in the corresponding quotient space. Theorem B1. If the characteristic vector field X satisfies codit.io (A ) and in the odd-dimesional case the Marrinet ideal (H) has the propert9 of zeros, then each of the lmear operotors Lx. Lt4 a, Lx.tî, is bjective, bas closed ima9e, and hos codinuous im,erse defined o the image. DISTIRIBUTIONS OF COIRANK I 2881 Pro@ By the Banach open lnapping theorem in Fréchet spaces it suffices to prove that each of the operators L,\- LH, and Lx. is injective and has closed image. The injectivity of the operators Lx, LH a.nd L.\-. follows from Proposition 3.2. (i), (il), (iii), respectively. The closedness of the image of the operator L follows h'om the global property of zeros inlplied bv Definition 1.5 by this property the image of LH coincides with the closed subspace C(M, S) c C(M) of hmctions vanishing on S. To prove the closedness of the image of the operators Lx and L,\-., we use Proposition 3.1. By Proposition 3.1. (i) the inmge of Lx coincides with the kernel of the continuous operator Vect(M) --+ V-'(A"-TM) defined bv -- X A]t, where F(A2TM) is the space of smooth scctions of the skew-symmetric product of the tangent bundle TM, with the C topology. Similarly, hv Proposition 3.1, (il) the inmge of Lx.n coincidcs with the kernel of the continuous operator l'ect'(S) -- F(A2(S:TM)) given by [Y] --+ IX A }'], where F(A"-(S:T3I)) is the space of smooth sections over ,_q of the skew-swmnctric pro(hlct of the t.angcnt bundle T3I (with the C topotogy) and [ ] denotes the equivalence çlass in the corresponding quotient space. Thc kernet of this op('rator is a closod subspace of l'cet ' (S: TIII) = Vectoe(l)/l'ectoe(M, S). The proof is complete. [] It is natural to ask if it is possible to replace the Coe-closeness of ,Zl to w0 in Theorems 1.2 and 1.4 by C-closeness with some r. Any attempt ai answering this question requires nlodification of Theorem B1, which was used in the proofs of Theorems 1.2 and 1.4. Proving Theorem 1.2. we had to show that the function AI. t [0, 1], does hot vanish at any point of M. In the proof of Theorem 1.4 we had to show that the functions BI, I [0, 1], do hot vanish ai points of S and the hmctions Ct, t [0.1], do hot vanish ai points of 31. The C-ctoseness of 1 to d 0 given as an sumption in Theorems 1.2 and 1.4 and the continuity of the inverse to the operators Lx, Ln and Lx.n allowed us to obtain the C-closeness of AI, BI and C, to 1. Of course, to «onclude that these hmctions do hot vanish, their C°-closeness to 1 woutd be enough. ç introduce the following topological chara«teristic of a linear injective operator L " C(M) COe(M) or L "COe(M) V««t(M) or L " C(S) Vect(S). Denote by m {0, 1.2 .... ;OE} the minimal m such that for anv s 0 the con- vergence to 0 of the sequence of sections L(f) in the C + topology implies the convergence to 0 of the sequence of hmctions f, in the C topology. This means that the inverse to L behaves hot worse than a linear diffcrential operator of order m. Note that bv Theorem B1 we have m(Lx), m(L), and ,n(Lx.) In many cases the nmnbers m(Lx), m(LH) and m(Lx.H) are finite and can be found or estimated from above, sec examples below. Tracing the construction of the functions AI, BI and Ci in the proofs of Theorems 1.2 and 1.4. it is easy to check that if these numbers are finite, then: 1. the C-closeness of AI to 1 holds provided that the 1-form 1 i8 close to 0 in the C topology with r = 2m(Lx) + 2: 2. the C°-closeness of BI to 1 holds provided that the 1-form in the C topology with r = 2m(Lx.H) + 2: 3. the C°-closeness of Ci to 1 holds provided that the 1-form d 1 i8 close to '0 in the C topology with r = m(Lx,H) + m(LH) + 2. Therefore in Theorems 1.2 alld 1.4 the Coe-closeness of W'l to 0 can be replaced by the closeness in a weaker topology, and we obtain the following result. 2882 B. JAKUBCZYK AND M. ZHITOMIISKII Theorem B2. In Theorem 1.2 the C°-closeness of wl to wo can be replaced by the CT-closeness with r = 2m(L¥) + 2. In Theorem 1.4 the Coe-closeness of wl to wo can be replaced by the CT-closeness with r = lnax (2n(Lx.H) + 2. m(Lx.H) + n(LH) + 2). Examples (n = 2k). 1. If X has no singular points, then it is clear that m(L_¥) = 0. Therefore the Coe-closeuess of w to w0 in Theorem 1.2 can be replaced by C2-closeness. Wc obta.in Theoreln 0.1. 2. If the 1-jet of X vanishes at no points of the nmnifold, then it is easy to prove that m(Lx) _< 1. Therefore the Coe-closeness of w to w0 in Theorem 1.2 can be replaced tri" C4-closeness. Examples (n = 2/,: + 1). 1. If (co0) is a Martinet distribution, i.e.. dH(p) ¢: 0 and X(p) ¢ 0 for an.v p .b' = {H = 0}, then it is easy to prove that m(LH) _< 1 and m(Lx.H) = 0. Therefore the C-closeness of CO 1 to CO0 in Theorem 1.4 can be replaced b.v C3-closeness. We obtain Theorem (/.2. 2. Assume that dH(p) ¢= 0 for any p S -- {H = 0}. Then the restriction of X t.o S is a. smooth vector field X]s on S. Assume that the 1-jet of Xs does hot vanish. In this case m(L) < 1 and m(Lx.H) <_ 1. Therefore the C-closeness of 1 to COe in Theorem 1.4 can be replaced by C4-closeness. ACKNOWLEDGMENTS While working on this paper we have profited fronl discussions with several colleagues. We are especially thankful for helpflll advice obtained from Pawet Domallski, Jean-Paul Gauthier, Pierre Mihnan and Richard Montgolnery. lAI [Arll] [BS] [BH] [DJ] [E] [1 [JP] [JZhI] [JZh2] t EFERENCES A. Agrachev, Methods of Control Theory in Nonholonomic Geometry, Proc. Internat. Congress of Math., Ziirich 1994, Vol. 2, pp. 1473-1483, Birkh/iuser. Basel, 1995. Mil 97f:58051 V. I. Arnold and Yu. S. Ilyashenko, Ordinary Differential Equations, in Encyclopaedia of Math. Sci. Vol. 1, Dynamical Systems 1, Springer-Verlag (1986). Mil 87e:34049; Mil 89g:58060 E. Bierstone and G. W. Schwarz, Continuous linear division and extension of C functions, Duke Math. Journal 50 (1983), 233-271, Mil 86b:32010 Il. L. Bryant and L. ttsu, Rigidity of integral curves of rank 2 distributions, lnventiones Math. 114 (1993), 435-461. MIl 94j:58003 H. Cartan, Variétés analytiques réelles et variétés analytiques complexes, Bull. Soc. Math. de France 85 (1957), 77-99. MIl 20:1339 P. Domafiski and B. Jakubczyk, Linear continuous division for exterior and interior prod- ucts (accepted to Proc. Aner. Math. Soc.). D. Eisenbud, Commutative Algebra, Springer-Verlag, New York 1994. MN 97a:13001 A. Golubev, On the global stability of maximally nonholonomic two-plane fields in four dimensions, Internat. Math. Res. Notices, 11,523- 529, 1997. MIl 98g:57043 J. W. Gray, Some global properties of contact structures, Annals of Math. 69, (1959), 421-450. Mil 22:3016 B. Jakubczyk and F. Przytycki, On J. Martinet's conjecture, Bull. Polish Acad. Sci. Math. 27 (1979), No. 9, 731-735. MIl 82e:58003 B. Jakubczyk and M. Zhitomirskii, Odd-dimensional Pfafiïan equations; reduction to the hypersurface of singular points, Comptes Rendus Acad. Sci. Paris, Série I t. 325 (1997), 423-428. MR 99e:58003 B. Jakubczyk and M. Zhitomirskii, Local reduction theorems and invariants for singular contact structures, Ann. Inst. Fouiner, 51 (2001). 237-295. MIl 2002e:58001 DISTIIBUTIONS OF CORANK 1 2883 [LS] [Ilg] [Mari [NIon] [Zhll W. Liu and H. Sussmann, Shortest paths for sub-Riemannian metrics on rank-two distri- butions, Mem. Amer. Math. Soc., Vol. 118 (1995), No. 564. MR 96c:53061 13. Malgrange, Ideals of differentiable functions, Oxford University Press, 1966. NIE 35:3446 J. Martinet, Sur les singularités des formes différentielles, Ann. Inst. Fouiner, Vol. 20. No.1 (1970), 95-178. NIE 44:3333 R. Nlontgomery, A survey on singular curves in sub-Eiemannian geometry, J. Dynamical and Control Systems, Vol.1. No.1 (1995), 49-90. NiE 95m:53060 R. Montgomery and M. Zhitomirskii, Geometric approach to Goursat flags, Annales de l'Instztut Henri Poincaré, Analyse Nonlineaire, Vol. 18, No. 4 (2001), 459-493. mie 2002d:58004 J. M. Ruiz, The Basic Theory of Power Series, Advanced Lectures in Mathematics, Vieveg, Wiesbaden, 1993. NIE 94i:13012 C. B. Thomas, Y. Eliashberg, and E. Giroux, 3-dimensional contact geometry, in "Con- tact and Symplectic Geometry" Publ. Newton lnstitute 8, Cambridge University Press, Cmbridge, 1996, 48-65. MR 98b:53026 M. Zhitomirskii, Singularities and normal forms of odd-dimensional Pfaff equations. Func- tional. Anal. Appl., Vol. 23, (1989), No. 1, pp. 59-61. ME 90i:58007 M. Zhitomirskii, Typical singularities of differential l-forms and Pfaflïan equations, Trans- lations of Math. Monographs, Vol. 113, Amer. Math. Soc,, Providence, RI, 1992. MR 94j:580{14 INSTITUTE OF NIATHEMATICS. POLISH fl.CADEMY OF SCIENCES. NIADECKICH 8. 00-950 Whrts.w. POLAND AND INSTITUTE OF fl.PPLIED lkIATHEMATICS, U'NIVEI:ISITY OF ,VARSAV. POLAND E-mail address: B. Jakubczyklimpan. gov. pl DEPARTMENT OF NIATHEMATICS, TECHNION, 32(100 IAIFA. ISRAEL E-mail address: mzhitechunix, technion, ac. il TRANSACTIONS OF THE AMER1CAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2885-2903 S 0002-9947(03)03289-6 Article electronically published on Match 14, 2003 WHEN ARE THE TANGENT SPHERE BUNDLES OF A RIEMANNIAN MANIFOLD REDUCIBLE? E. BOECKX ABSTRACT. \e deternfine ail RiemaImian manifolds for xvhich the tangent sphere bundles, equipped with the Sasaki metric, are local or global Riemann- ian product manifolds. 1. INTRODI¢CTION When studying the geometry of a Riemmmian manifoM (3/. g), it is often useflfl to relate it to the properties of its unit tangeut sphere bmMle Tt3I. hl earlier work, we have been primarily interested in the geometric properties of T3I when equipped with the Sasaki metric gs. This is probat»ly the sinll)lest possible Rie- mamfian metric on T3I and it is completely determined by the nletric 9 on the base manifold ]lI. In this way, we bave obtained a. nUnlber of interesting charac- terizations of specific classes of Riemmmian manifolds. We refer to [2], [5]. [6], [7] and the references therein tbr examples of this. Also tangent sphere bundles T,.M with radius r different from 1 and equipped with the Sasaki metric bave been stud- ied recently ([9], [10]). The geometric properties of these Riemmmian manifolds may change with the radius. Sec [9] for an example of this. Of course, other tiemannian metrics on the tangent bundle and on the tangent sphere bundles are possible. Of these, the Cheeger-Gromoll metric 9cc may be the best known. However, for tangent sphere bundles, this specific metric yields nothing new. since (T,-M, gcc) is isometric to (T,./4i-4M, gs ). The isometry is given explicitly by p: Trl]l -- Tr/ lvq-l[: (X, t) v- (X, u/V + r2). It is a.n interesting geometric problem to determine when a tangent sphere lmn- dle, which we alwoEvs consider with the Sasaki metric in this paper, is reducible, i.e., when it is locallv or glol)ally isometric to a Riema.nnian product manifold. To our surprise, we could not find anv results in the literature concerning this ques- tion. Nevertheless, knowledge about reducibility could help to deal with geometric questions about, tangent sphere bundles, lu [4] for instance, we use it in an essen- tial way to determine all unit tangent sphere bundles that are semi-symmetric, i.e., for which the curvature tensor at each point is algebraically the saine as that of some symmetric space. Actually, that problem was the inspiration for the present article. As concerns the local reducibility of tangent sphere bundles, we prove here the following. Received by the editors November 11, 2002 and, in revised form, January 21. 2003. 2000 Mathematics Subject Classification. 53B20, 53C12, 53C20. Key words and phrases. Tangent spheïe bundle, Sasaki metric, reducibility, Clifford structures, foliations. @2003 American Mathematical Societ.y 2886 E. BOECKX Local Theorem. A taTgent sphere bmdle (T,.]iI, gs), r > O. of a Riernannian manifold (M ', g). >_ 2, is locally reducible if and mdy tf (M, g) bas a fiat factor. i.e.. (lI,g) is locally isometric to a product (M',g') × (llk, g0) where 1 _ k _ n and go denotes the standard Euclidean metric on II . The corresponding global version reads as follows: Global Theorem. Let (]il '.g). n >_ 3. be a Riernannian rnanifold and suppose that (TM. gs) is a global Rienmnnian product. Then. (]il, g) is either fiat or it is also a global Riemamia product, with a .fiat factor. Coversel.q, if(M, g) is a global product space (M', g') × (F , go) where 1 <_ k <_ n ad F is a co.nnectcd ad sin@ly connected .fiat space, then (TM, gs) is a global Riemannian product, also with (F, go) as a .fiat factor. In vicw of thc commcnts above, thcse results remain valid if we consider the tangent sphere bundles equipped with the Cheeger-Gromoll metric. This article is organized as follows. After giving the necessary definitious and formulas concerning tangent sphere lmndles, we show in Section 3 that onlv two types of decomposition for TM are possible: a vertical and a diagonal one. The special form of the curvature of (TrM, gs) for xrertical vectors is crucial here. In particular, the saine procedure does hot go through for the tangent bundle TM. Sectiou 4 deals with the diagonal case. We find that a diagonal decomposition gives rise to a Clifford representation via specific curvature operators. As a result. only base maififolds with dimension 2, 3, 4. 7 or 8 could possibly adinit diagonal decolnpositions. The diffelent dimensions are then handled separately. It turns out that diagonal decompositious can only be realized for a fiat. surface as base space. The general situation with a vertical decolnposition is treated in Section 5 and leads to the Local Theoreln above. The final section is devoted to global cousiderations. 2. TANGENT SPHERE BUNDLES We first recall a few of the ba.sic facts and formulas about the tangent sphere bun- dles of a 1Riemannian manifold. A more elaborate exposition and further references can be found in [51 and [9]. The tangent bundle Tll of a Pdelnannian lnanifold (]il. g) consists of pairs (:r, u) where 3c is a point in M and u is a tangent vector to M at .r. The mapping 7r : TM -- M: (:r,u) - x is the natural projection from TM onto M. It is well known that the tangeut space to T]II at a point (a-, u) splits into the direct sure of the vertical subspace I'T]II(z,, 0 = kerr.l(,, 0 and the horizontal subspace HT]II(,, 0 with respect to the Levi-Civita connection H T ]I I( ,, 0 For w T,]II. there exists a unique horizontal vector w HTM(z., 0 for which 7r.(w ) = w. It is called the horizontal lift of w to (x, u). There is also a unique vertical vector w I'T]IIt.,O for which w(df) = w(f) for all fimctions f on ]iI. It is called the vertical lift of iv to (x, and HT]II(,, 0 and VT]II(,,O, respectively. Hence, everv tangent vector to T]II at (:r, u) can be written as the sure of a horizontal and a vertical lift of uniquely defined tangent vectors to ]Il at :r. The horizontal (respectively vertical) lift of a vector field X on M to TM is defined in the saine way bv lifting X pointwise. Further. if T is a tensor field of type (1, s) on M and X1,..., X-I are vector fields on ]il. then we deuote by T(X1 ..... ..... X_) " the vertical vector field on TM WHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2887 which at (ce, w) takes the value T(Xlc,...,lv .... , Xs_ix) v, and sinfilarly for Ihe horizontal lift. In general, these are hot the vertical or horizontal lifts of a vector field on/ll. The Sasaki metric gs on T/iI is completely deIermined by 9s(x h, ) = 9s(x v. ") = 9(x, ) o , 9s(x . v) = 0 for vector fields X and }" on/il. Our interest lies iu the tangent sphere bundle T,./i1 of some positive radius r, which is a hypersurface of Tal consisting of ail tangent vectors lo (/il, g) of lenglh r. Il is given implicitly by the equation g,(«, u) = r 2. A unit normal vector field N to T/iI is given b.v the vertical vector field "/r. We see that horizontal lifts to (:r, u) TM are tangent fo T/II, bul vertical lifts in general are not. For that reason, we define the tangential lift u , of u' T.Cl to (x, u) TI by 1 wt = wv - 7 g(w.u)N. Clearly, the tangent space lo T5I at (x,u) is spmmed bv horizontal and tmgential lifts of taugent veclors to M at x. One defines the tangential lift of a vectorfleld X on/iI in the obvious ,va.v. For the sake of notational clarity, we will use 2{" as a shorthmd for X - g(X, u)u. Then .\' = .(v. Further, we denole by VT,.kI he (u - 1)-dimensional distribution of vertical tangent vectors to TrM. If we consider THI with the metric induced from the Sasaki metric 9s of T/i1, also denoted by 9s, we turn TrM iuto a Fliemannian manifold. Its Levi-Civita Colmection Ç is described completely by () 1 V x, I "t = o(I",,z)X t, Ç\.hY h : _ (H%A')y) h, 2 1 = (Vxr)'+((,,.)x) h, - (Vx}')h I((X,}'))' for vector fields X and Y on/il. It.s Riemann curvature tensor/) is given bv (2) (X h, Yh)Zt (X h, Yh)zh for vector fields X, Y and Z on/il. (See [9].) 2888 E. BOECKX 3. TWO TYPES OF DECOMPOSITION Let (/l.g) be a Fliemmmial mmlifold of dimension , k '2 and suppose that its tangent sphere bundle T,.M is (locally) reducible, i.e., (TM, 9s) (/1,gl) x (M2, 9)- A point (.r, u) in TM corresponds to a couple (p. q) I1 X 2lh, and the tangent space T(,,)TkI can be identified with TpI1 @ q.lI2. In the sequel, we will write T(,)kI and T(,)kI for TpI1 and TqM2, considered as subspaces of T(,)TM. in order not to make the notation too cumbersome. Suppose first that, at a point (.r, u) of TkI, the tangent space to one of the factors, say to I1, contains a nonzero vertical vector X t, X TkI and X K u. Since we have a Riemannian product, the curvature operator (U, V) preserves the tangeut spaces to both factors for ail vectors U and V tangent to TtM. In partiçular, it follows that 1 (}", xt)x t = r (9(.¥, X) }-' - g(X. }')X t ) C T(.u)3ll for all vcct.ors } T«;I. As a consequence, I'T5I(.) C T(.u)M, and lI is at lcast (,- 1)-dimcnsional. Hencc, (f at a point 4E;l one of the factors contains a nonz«v vertical ve«tor, if «ontains fb« complete vertical distribution al that point. call the decomi)ositiou vertical at (a'. u) in such a situation. Note that this is the ce as soon m{dim fil1, dira M2} > n. Indeed. if dira fill > , then - dim(l'E,lt,u ) + > (., - 1) + - (2,, - 1) = 0. So, the only possibility %r the decomposition hot to be vertical at (z. u) is that dira I1 = , dim.12 = - 1 {or conversely) and neither factor is tangent to a vertical vector. X call this a diagonal decomposition at (x, ). The major part of the sequel will be devoted to the diagonal case. Using a purely infinitesimal (i.e., pointwise) approach, we show that a diagonal decomposition is only possible in one specific situation. Afl.erwards. we study the case of a vertical decomposition. 4. DIAGONAL DECOMPOSITION 4.1. A suitable basis. In this section, we consider a diagonal decomposition Tr]l 1I1 x M.2 at (x, u) with dira/I1 = "n and dira/I2 = - 1. For dimen- sional reasons, we bave dim(T(:,,)3ll HT3I(,.,)) > O. Let Xn TII be a unit vector such that Xn h is tangent to Il[ 1 at (x. u) and extend it to an orthonormal basis {X .... ,X,} of T«M. If rc.(,,,)(T(,,,)M1) ¢ TM. then there nmst be a vertical vector tangent to/I1 at (x. tt), contrary to the hypothesis. Hence, there exist well-defined vectors Y ..... }-t orthogonal to tt for which .\-h + tlt ..... Xn_l h ÷ t'n_l t and X, h are tangent to 1[ 1 at (x.u). Clearly, they form a basis for T(æ.)M. though uot in general an orthonormal one. Moreover. {Y1 ..... t;_, u} is a basis for TM too. Otherwise, there would exist a nonzero vector t" TII, orthogonal to tt and to Yi, i = 1 ..... - 1. But then t "t would be orthogonal to X, h and to Xi h + E t, i = 1 ..... - 1. and hence WHEN ARE TOENGENT SPHERE BUNDLES REDUCIBLE? 2889 would belong to (Tla:,u)11Ii) - = T(a:,u)111.2, contrary to the hypothesis that kI2 has no vertical tangent vector. Next, consider the (n- 1) x (,,- 1) matrix a = (g,(I: ))i4= ..... _. Since this nmtrix is symmetric and positive definite, it cm be diagolmlized bv a suitable orthogonal transformation: p¢pt = diag(A2 .... , kn_l 2) where P = (Pij) 0(,- 1) and A, > 0 tbr i = 1 .... , - 1. Ifwe p/lt j=l j=l fo = ..... - . thon oth {X5 .... or{honormal bases for T,M. Further. thc vectors 2, + ,L'= ,, (a logether wi{h Xn span the {angenl spa«e {o kI at (x. u) and these xwctors are pairwise orlhogonal. The tangent space to orthogonal vectors A,çi -«, i= 1 ..... t-1. Finally, we show that all the gs(/(U, V)W, T) = 0 at (x, u) to/111 and another one is tangent it follows that Using the expressions (2) for the condition numbers Ai are equal. To do this, we use that as soon as one of the vectors involved is tangent to M» In particular, for all i, j, k. l = 1 ..... n- 1. curvature tensor/ of (TII. gs), this lea.ds to the 0= (g((2»_L)-.fi)-g((,)-i)-ï»(.)-L)) ,,2 - Switching the indices i and j. as well k and I, we find Using the symmetries of the curvature tensor, it then easily follows that Ai 2 = Ai 2 Summarizing, we have Lemma. If Thl ,I x M2 is a diagonal decomposition et (x, u) u,ith dira 5I = and dimM2 = n - 1. then there exist orthonormal bases {X,...,X,} and is given by X1 h + A)] , .... X._ + A)_ t, X,, h and an orttogonal basis for T(z,u)M2 is given by 2890 E. BOECKX ReTnark 1. The number ) has a clear geolnetric meaning. Take a nonzero vertical rt--1 vector U at (x, t)" U = i= and a nonzero vector V tangent to M2 at (x, u): = i= fli (AXi h - t). The angle between thc two vectors h cosine given by «o() = - E By the Cauchv-Schwarz inequality, we bave 1 < cos(U) < 1 fi + A: - fi + A: with equality if and only if (a,,... ,a-l) and (fil,---,Ç-I) are proportional. conc]ude that the angle t)etwccn lTrI(z,) and T(z.)II2 is such that cos 1/ + A 2 or tan = A. So, A determincs the angle between 1 "ThI and Iee at (x, (and hcnce also between I'T[ and hI at that point). Hemark 2. Actually, we can give a st.ronger formulation of the lemma. To see this, consi(tcr the mapl)ing " T¢.)M I'TM(z,)" X h + l-t l-t. Clearly, this ma.l)l)ing is linear an(l onto-onc on (Xh) . restrict to (X,,h)x T(x.)SI and dcfine the linear mapping A" a X x" Y An.(x,)(n ) whcre, as 1)cforc, " TrM M is the natural projection map. Since .a = .(.)(,-')')= .(.)((.X', + Y=*)/) = the map A is an isometry from u to X . It associa.tes to a vector X, orthogonal to X, the unique vector ', orthogonal to u, such that Xh+ Al t is tangent to at (x, u) (or such that AX h - yt is tangent to ,I2 at (x, a)). So, in the lemma, we can actual]y choose an arbitrary orthonormal basis {X1,..., X-I} of X (or, alternatively, an arbitrarv orthonormal basis {) ..... )_} of u). will use this possibility in the subsequent subsections. The vectors X (up to sign) and u, on the other hand, are determined geometrical]y. 4.2. Curvature conditions. Since (T,.M, gs) is a (local) Riemannian product, all the expressions of the form (U. V)W are zero when U is tangent to hl and W is tangent to Iee at (x, u). Using thc curvature fornmlas (2), this leads to a number of curvature conditions for the manifold M. Wc list some of these now. om now on, indices i. j, k and l belong to {1,... ,n - 1} unless stated otherwise. The tangential and horizont al components of (X h. ) t) (AXkh _ t) give fise to 2 (3) n(.X', ÆX')Y) - g(n(.X, X»), ) = while (X h, ,kÇ-h)(AX h -- tt) = 0 leads to 4 (5) (6) WHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2891 Considering/(X/h + At'i t. }'jt)(/xkh -- t) : 0. We obtain 2 (7) 4 = (8) 2(vx,)(..))x+2(),)x+ (..))(.)x Finally, from (Xi h + t, Xjh)(Xk h _ t) = 0, we derive (9) 2A These conditions can be rewritten in an casier form. To start, we take the inner product of (3) with t}. This gives 2((.\',,, .\-) k;., ) This is equivalent to = -(t(, )t(, i)x,.\). (11) 2R(k-, t))X + R(,t))R(u.t))X, =-g(R(a. Yj)X,,R(u,t))X,)X.. By interchanging the indices j and l iii this expression and adding both formulas, respectively subtracting theln, we get (12) R(u,I-)R(u,I))X,, + = -2(n(. ? )x. (. i)x,) x, (13) R(u, t)-)R(, l))X. - R(u. t ))R(u, t ))X. = 4R(I ). t-)X. Substituting (11) iii (4), we filld the simpler form (14) 2A(Vx.R)(u. Next, we substitute (3) in (5) to obtain Taking the inner product with t, we get = = 0 2892 E BOECKX by (12). Hence, (15) (V.\-kt)(_¥n,.¥j)tt = 0 or. equivalently, Sul)st.itutiug (14) aud (15) lu (6), we find (16) (Vx.R)(. i)xn =0. 1 ._ ( R(. Y )X,.. R( .. Y )X,,) = R(.\'n, Xj)X + 2(,,. (X., - (-,,. R(X» X).)X + lu order to rewrite (7). we proceed as with (3): we take the inner product with aud we use curvature properties to obtain 4 (1) 2R(,)X, + :(..Y):(,.'4)X = (»Xç- X). (Note that we also ueed (11) to know that the left-hand side in (17) is orthogonal t.o X.) Again switching the iudices j and I and adding and subtracting the two fi)rmulas, we get 4 (19) R(u,))R(u.}})X,- R(u,}})R(u.))X, = 4R()},))X + SM)st.itutiug (]7) and (19) in (8), this rcduces to (2o) (v x,R)(, ))x = or equivalently, via (15), to (21) A(Tx, )(Xk, -\')u = ,( - 1) r (&X - X), 2( - - 1) r2 ((ik Yl -- (ilYk). It is now easilv verified that (9) is a consequence of the above formulas. As to (10), usiug (17) and (20), it simplifies to (2) 4EE(x. x)x + R(, (x, x-))x- (. (%., x))x In the rest of this section, we wi]] on]v need the Nrmu]as (12), (13), (16), (18), (19) and (22). 4.3. Clifford structures. Putting j = 1 in (12) and (18), we see that 4 Since (, ) is a skew-symmetric operator, the nonzero eigenvalues of R(u. must bave even multiplicity. Hence, *if n is even, the eigenvahm -4/r 2 has even multipli«ity n - 2 on {Xj, X} . Hence. the eigenvalue corresponding to X must be zero. This implies that n**,Y))X = 0 for = .... ,, - . By (3), so n()),)X = 0 for j, k = 1 ..... n - 1. conclude that X,, belongs to the nullitv distribution WHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2893 of the curvature tensor R. |n this case, the conditions (12), (13) and (16) are trivially satisfied; if n is odd, the eigenvalue -4/r 2 has odd multii)licity n- 2 on {X, X,} ±. So, the eigenvalue corresl)onding to X, nmst be -4/r 2 as well. Hence, it follows that IR(u. }))X,I 2 = 4/r 2 for j = 1 ..... n - I. Bv Remark 2, we even have IR(u, Y)X,I 2 = 4/r 2 for every mfit vector 7t" orthogonal to u. Po- larizing this identity, we obtain 9(R(u. I')X,, R(u. Z)X,) = (4/r 2) 9(Y, Z) for all vectors V a.n(l Z orthogonal to u. In l)articular, the right-hand side of (12) equals -(85jt/r2)X,. In this case, conditions (12) and (13) are included in (18) and (19) if we allow the index i to I)e n. Next, we put i = j -¢ l in (18). Since R(u, 7t))Xj = 0 (this follows ri-oto R(u,I'))2Xj = 0), we obtain R(u.}'j)R(u,})Xj = (4/r2)Xt. Avplying the op- erator R(u, I)) on both sides, we have or, equivalently, Since the right-hand side of this expression vanishes both when n is odd and when is even. we conclude (23) for j, l = 1 ..... n- 1. Ve are now ready to discover Clifford representations in our fornmlas, in partic- ular in (12) and (18). First, consider the case when n is even. For j = 1,..., - 1, define the operators Ri acting on V" = T,]II by T R, = _ (, }i) - (x,, .)x, + (x, .)x, wllere (-,) = gc. In particular, it follows that R,Xi = \'n, RX = -Xi and RXj = (r/2)R(u, ;)X-, j ¢ i. Clearly, R, is a skew-symmetric operator and Ri 2 = -id. 2894 E. BOECKX For i ¢ j ¢ k ¢ i, we calculate: (Ri o Ri + Ri o Ri)Xn = -RiXj - RjXi =-ï-(R(u,Y)Xj+ R(u,}))Xi)=0. (by (23)) 2 (Ri c Ri A- Ri o Ri)Xi = Ri(/(u, })Xi) A- T 2 = --f R(u, Yi)R(,Yj)Xi- Xj =0 (by (18)) ,. r R(u, E)Xk) + (,5){(,;-)X-g(R(,})X,Xj)X}) T 4 =0 (by (s) la (23)). So, for i,j = 1,..., - 1. the operators R, satisfy R o R + R o R, = -2 iid and they correspond to a Clifford representation of an (n - 1)-dimensional Clifford algebra on an n-dimensional vector space. It is well known (see, e.g., [1] or [3]) that a given real Clifford algebra, say of dilnension m, has onlv one (if m - 3 (mod 4)) or two (if m =- 3 (mod 4)) irreducible representations and that the dimension n0 of the corresponding irreducible Çlifford module is completely determined by m. This relationship is given in the following table. rn 8p 8p+l 8p+2 8p+3 8p+4 8p+5 8p+6 8p+7 no 2 4p 2 4p+I 2 4p+2 2 4p+2 2 4p+3 2 4p+3 2 4p+3 2 4p+3 For a reducible Clifford module, the dimension is a multiple kno of the number n0 corresponding to the appropriate Çlifford algebra. In the present situation, we bave m = n- 1 and kno = t for even . Therefore: if = 8p: 8p = k24p- and hence p = 1, k = 1 and n = 8; if n = 8p + 2: 8p + 2 = k24p+l and hence p = 0, k = 1 and n = 2; if = 8p + 4: 8p + 4 = k2 4p+2 and hence p = 0, k = 1 and n =- 4; if = 8p + 6: 8p + 6 = k24p+a, which has no solutions. Next, suppose that is odd. Now, we define operators Ri, i = 1 ..... n - 1, acting on V n+ = TxII NXo T R; = n(, y)- (Xo, .)x, + (x,,-)x0 WHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2895 where ( -, .) = .q. @ go with go(aXo, bXo) = ab. Precisely as before, we show that 1 i o lj + Rj o 1 i = -25ia id for i, j = 1, .... n - 1. So, we bave again a Clifford representation, this time with m = n - 1 aud/,"n0 = n + 1 for odd n. Therefore, by the table al)ove: if n = 8p + 1: 8p + 2 = k2 4v altd hence p = 0. k = 2 and n = 1: if,=Sp+3:8p+4=k2 4v+2andhencep=0,k=land=3: if n = 8p + 5: 8p + 6 = k2 4v+a, which has no solutions: if = 81»+ 7: 8p + 8 =/,'2 4v+a and hence p = Il. k = 1 and = 7. We conclude from this subsection that diagonal decompositious can only occur when the base manifold has dimension 2.3, 4, î or 8. (The case n = 1 is irrelevant, since then T,.M has dimension equal to one and no decoinpositions exist.) 4.4. The relnaining dimensions. Case n = 2. In this situation, we have a two-dimensional manifold for whiçh the mfllity vector space of the curvature tensor is non-trivial. This implies that the curvature tensor is identically zero and the space is fiat. Conversely, sin«e any tangeut sphere tmndle of a fiat surface M2(0) is a fiat three- dimensi(mal space, a diagonal decomposition actually exists around each point (x. u) of TM'(O). Note, however, that we also bave T]II2(O) - M")(O) x S(r) with {x} x S(r) _ a--l(x). So. TM'(O) also admits a vertical decomposition. Case n = 3. Let Xa 1)e the unique unit vector (Ul) to sign) such that Xa h is tangent to kl at {x, u). Pick a Ulfit vector X orthogonal to Xa and let be the corresponding unit vector orthogonal to v {i.e., .¥h +A)-t is tangent to .iii). l¥oln the comments at the beginlfing of Subsection 4.3. we know that (r/2)R(u.))Xa is a unit vector, which is moreover orthogonal to X1 and Xa. So, we obtain an orthonormal basis {Xt,X2, Xa} by defining X2 to be X: := (r/2)R(u.'a)Xa. Let t) be the corresponding unit vector orthogolml to u and 1". (Since each 1 is fixed together with its correspouding Xi, we will hot luentioli this explicitly anvlnore in what follows.) Using the properties of the operators R( u, ') and (v. ), we then deduce that (24) R(u, Y1 )x = 0, R(u, Y )x = - -. x3, /(, ?)x, = 2_ xa, R(u, ?)x2 = o, and from (13) and (19) it follows that R(tt "1.)X 3 - 2_ Next, we COlnpute R(X, Xj)X, i, j. k = 1.2.3, ffonl the equalities (16) and (22). writing R(u,R(Xi, X)u)Xa as Y'9(R(u, Yz)Xi, Xj) R(u,})Xt. and using (24)and (25). This gives (26) 1.2 R( X1, 1. R(X,, X3) 1.2 R(X2, X3) Xa X2 Xa -AX AX 0 -CX3 0 CX 0 -CXa CXa where A = (,4 _ ,2 q_ 1)/),2 and C = (aA 2 + 1)/ 2. (25) t(Y,)xl=-x2, (,)x.=xt, (Y,)xa=o. 2896 E. BOECKX Since both {X1, X2, X3} and {¥], t), u/r} are orthonornial bases for Txll, there is an orthogona] matrix Q = (qij) 6 0(3) such that X., = Q Y., Changing Xa to -Xa if necessary, we ma,y even sut»pose that Q 6 SO(3). Then R(Xt, A'3) = (qlqa2 -- qt2q31) H(}], }')) + qllq33 --ql3q31 ?- + q12q33- q13q32 R(}), = -q3 R(Y1, t;_) - q2___ R(u, ti) + q__l R(u, T F If we let both sides act. on X1, ¥._, and X3 and if we use (24), (25) and (26), we find that q21 = -C/2. q2 =0, q3 = 0. Since Q S0(3), it follows that qe2 +qe22 ÷q23 2 = 1 and hence 1 = (3A2+ 1)/2A 2 or A - + 1 = O. which is a contradiction. Hence, no three-dimensional manifold adnlits a diagonal decomposition of its tangent sphere bundles at anv point. Case 11 = 4. Let X4 be the unique unit vector (up to sign) in the nullitv distribu- tion of R,. Take two mutuallv orthogonal unit vectors X and X.,,_ perpendicular to X4. Since (r/2)R(u, Y1)-\'2 is a unit vector and orthogonal to X1, X2 and X4, we can define X3 := (r/2)R(u, t])X2. From the properties of file operators R(u. i = 1,2, 3, it follows that (27) r R(,,, "1) ," R(u, t&) ,'R(u, t5) X1 X2 Xa X4 0 2Xa -2X.2 0 -2X3 0 2X 0 2X -2X 0 0 Next, we decompose Xa with respect to the basis {tq, t.), t, u/r}: X4 = qt] + q2I') + q3} + qa -, q + q2 + q32 + q42 = 1. Then R(u, Xa) = ql R(, }'1) ÷ q R(u, Y.,_) + q3 R(zt. }5)- Since Xa belongs to the mfllity distribution of R, this operator vanishes identically. By (27), we nmst bave q = q2 = q3 = 0. Hence, Xa = :t:u/r. But this is impossible, since u clea.rly does hot belong to the nullity distribution. So, also for four-dimensional mallifolds, a diagonal deconlposition of its tangent sphere bundles does hot exist at anv point. Case n = 7. The argtlnlent for n = 7 goes along the Saille lines as that for n = 3. but it is more involved technicallv. Again we start with the trait vector Xr, uniquely VHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2897 deternfined up to sign. su«h that Xr h is ta.lgent to 2I, and with an arbitrary unit vector X, orthogonal to X7. The unit vector X2 := (r/2)R(u, }',)X7 is orthogonal to both X aud X7. Then it follows t.hat R(, ;)x, = o, R(u, }))X = - Xr, ,ï, R(u. }')X2 - 2 X7, R(.u. )x2 = o, R(-u, Y)X7 = - X2, R(u, ')Xr - " -- -TX. Note that R(u. Y1) aud R(u, t?) preserve spal,{Xl,-\'2, X7}, hence by skew-sym- metry also its orthogonal complemeut. Next, take a unit vector X4 orthogonal to X1, X.2, Xr and define the unit vectors Xs := (r/2)R(u,t))XE aud X := (r/2)R(u, t'I)X4. Then Xs and X are ah'eadv orthogoual to X, X2, X4 and Xr. Further, £2 i-2 -- g(R(u,}])R(u,}'2)X4, X4) 4 £2 (28) = g(R(u.I))R(u,}i)X4,X4) (by (18)) r 2 -- g(R(u, })X4, R(,u, )X4) 4 = -{Xs, and X5 and X6 are mutually orthogonal as well. Finally, sin«e R(a, ]q)X5 is or- thogoual to X, X2, Xs, Xr and we mav defil,e X3 := (r/2)R(u, I'I)X. In this way, we have defined ml orthonormal basis {X1 ..... Xr}, and the actionb of the operators R(u, }), i = 1 ..... 6, can be COUlputed explicitly in this basis using the properties (12), (18) and (2a) above. We obtain r R(,, \) r R(u, " R(-u, r R(u. r R(-. X X.2 X3 X4 X5 X6 Xr 0 -2Xr -2X5 2X6 2X3 -2X4 2X2 2Xï 0 2¥6 2X5 -2X4 -2X3 -2X1 2X5 -2X6 0 -2X7 -2Xt 23£2 2X4 -2X6 -2X5 2X7 0 2X2 221 -2X3 -- 2X3 2X4 2X1 - 2X2 0 - 2X7 2X6 2X4 2Xa -2X2 -2X1 2Xr 0 -2X5 2898 E. BOECKX Next, ve calculate R(}]-, }-)Xk from (13) and (19): X1 X2 23 X4 X5 26 X7 --222 2X 2X4 -2Xa 226 -2X5 0 -2Xa -224 22 222 0 2XEE -2X4 223 -222 2X1 227 0 -2X5 -225 -2X6 0 -227 221 222 224 -226 2X5 -2XEE 0 -222 221 2Xa 224 -2Xa 222 -22 227 0 -225 -223 -2-\'4 221 222 0 -227 2\'6 2X6 -225 -2X7 0 22 -22 223 -225 -226 0 227 22 222 -2Xa 2X2 -22 -224 223 226 -225 0 0 227 -2-¥5 -226 223 224 -222 227 0 -226 2X5 -224 223 -2XI 227 0 226 -225 224 -223 -2XI 0 -227 -225 -226 2X3 224 222 -2Xt 224 -223 -226 225 0 Using (16) and (22), ve ca,, now compute the «,r'«ature components R(Xi, 2j)Xk for i,j,k = 1 ..... 7: r 2 R(X r 2 R(X1 r 2 R(X r 2 R(X1 r 2 R(X1 r 2 R(X1 r 2 R(X2, r 2 R(X2, r 2 R(X2, r 2 R(X2, r 2 R(X2, r 2 R(X3, r 2 R(X3, r 2 R(X3, r 2 R(X4, r 2 R(X4, ,,.2 R(X4, r 2 R(Xs, r 2R(X6, X1 22 X3 X4 X5 X6 X7 -A22 AX 224 -223 226 -225 0 - B23 24 BX - 22 0 0 0 -B24 -X 3 22 BX 1 0 0 0 -BX5 X6 0 0 BX -X2 0 -BX6 -X5 0 0 22 BX 0 -CX7 0 0 0 0 0 CX -X4 -BX3 BX2 X 0 0 0 23 -BX4 -Xt BX2 0 0 0 -X6 -BX5 0 0 BX.2 X 0 25 -BX6 0 0 -X BX2 0 0 -ex 7 0 0 0 0 C. 2 222 --221 -AN4 A23 226 -2X5 0 0 0 - BX5 26 BX3 -X4 0 0 0 -BX6 -25 24 BX3 0 0 0 -CX 7 0 0 0 CX 3 0 0 -X6 -BÆ5 B24 X 3 0 0 0 25 -BX6 -23 BX4 0 0 0 0 -CX7 0 0 CX_ 222 -2X 2X4 -223 -AX6 AX5 0 0 0 0 0 -CX7 0 CX5 0 0 0 0 0 -CX7 CX6 where A = (A 4 - A 2 + 1)A 2, B = (A 2 + 1)2/A 2 and C = (3A 2 A- 1)/ 2. We now show that the tables above are incompatible. To see this, we relate the two orthonormal bases {X ..... 27} and {} ..... Y6, u/r} by an orthogonal WHEN ARE TANGENT SPHERE BUNDLES REDUC1BLE? 2899 transformation. Let Q = (qij) 0(7) be such that Putting Q, := qi.qj - qqj, we thon have the equality 6 6 ij k<l=l k=l No, and 2 = r g(R(X,X2)X*,X4) 6 6 k</=l k=l = (QI - Qàï + Q) u)X3, X) 12 2 : .2 g(I(XI,X2)X5,X6) _ 2(Q22 ÷ Q3142 _ Q56)- This implies that Q] = 1. Now, using the Cauchy-Schwarz inequality and the fact that Q is orthogonal, we find that 1 = QI = qllq2-. - Hence, ail the inequalities must be equalities. In particular, we have q13 .... ql7 = q23 .... = q27 = 0 and consequently X =cosO l +sinO )), X.2 =q(-sinO1Yl +cosO I) where ci = :t:1 and 01 is some real number. In a similar way, we can show that Q = Qs566 = 1 and that X 3 --- COS 0 2 }] ÷ sin 02 X5 = cos03 t' + sin03 Y6, X4 = e2(-sin02 l + cos02 X6 = 3(- sin 03 } ÷ cos03 Y6)- As a consequence, we also have X7 = cuir, e = -1-1. Using the tables above, we find that 0 = r e R(X, XT)X3 = -e (cos0 r R(u, }'1)X3 ÷ sin01 r R(u, = 2e(COS01Xs - sin01 X6), which gives a contradiction. So, also seven-dimensional manifolds cannot bave a diagonal decomposition for their tangent sphere bundles at anv point. Case n = 8. This case is treated as the case n = 4, but the appropriate choice for the basis {X1,..., Xs} requires a little more tare. Let Xs be the unique unit vector (up to sign) in the nullity distribution of R and take two arbitrary unit vectors X and X2 that are mutually orthogonal and perpendicular to Xs. As before, we 2900 E. BOECKX define X3 := (r/2)R0*, })X2, which is a unit vector orthogolml to X1, X2 and It follows that R(u. }))Xs = 0 for i = 1.2, 3, and n(.. }i)xl =0. R(U. l.))X1 -- 2 3, R(u, Ya)X1 = 2_ Because they are skew-symmetric, the operators R(u, }i), H(u,}:z) and R(u.};) also preserve l I = { X 1, X2, Xa, Xs } ±, and on t his space t hey ant i-conmmt e by (18). It is easv to check that the operator (ra/8)R(u, I])R(u, I))R(u. }5) is symmetric on II" and that it squares to the identity on Il . Hence, it bas a basis of eigenvectors corresponding to the eigenvalues +1 and -1. Let Xa be a unit vector in Il" such that r a R(u, Yt)R(u, Yz)-/ï'(u, }5)Xa = 8eX4 where e = +1, and define r/(. };)X. Clearly, X5, X0 and X7 are unit vectors orthogonal to X1, X2, X3. X4 and Xs. A comlmtation similar to (2) shows that they are also orthogonal to one another. So, we have an orthonornml t)asis {X1,...,Xs} for T3I. It is now possible to compute explicitly the action of R(u. ), i = 1 ..... 7. from the condition (18). VVe get /(, i) r R(u, Y2) -/(u.}) .,-/(.. ;) X X2 X3 X4 X5 X6 X7 Xs 0 2-\'3 -2X2 2X5 -2Xa -2eX7 2e-\'6 0 -2X3 0 2X 2N6 2eX7 -2Xa -2eX5 0 2X2 -2X1 0 2X7 -2eX6 2eX5 -2X« 0 --2X5 -2X6 -2X7 0 2XI 2X2 2X3 0 2X4 --2¢-\'7 2cA'6 -2.Y1 0 -2e-¥3 2e-\'2 0 2eXz 2Xa -2eX5 -2X2 2eXa 0 --2eX1 0 -2eX6 2eX5 2X4 -2Xa -2eX 2eX1 0 0 Next, decolnpose X8 with respect to the basis {I],..., I), u/r}: Xs=qll]+---+qE})+qs-. ql 2+.--+qs = 1. Since X8 belongs to the nnllity distribution of the curvature, we have 7 o= t(,x) = , q t(.) and ffom the table above we deduce ql = .... q7 = 0. Hence. XS = Tu/r, but this is impossible since u does hot belong to the nullity distribution. So, also in the eight-dimensional case, a diagonal decolnposition of the tangent sphere bundles does not exist at even a single point. Remark 3. The operator r 3/(u. ]/)]:(u,})/(u. Y3) acts as 8eid on the vector space spanned by X4, X5, X6and Xr, as is seen easilv froln the previous table. The two different cases, e = +1 and e = -1, correspond to the two non-equivalent irreducible Clifford representations of the seven-dimensional Clifford algebra. \VHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2901 5. VERTICAL DECOMPOSITION Now, we sui)pose that we have a vertical decolnposition T,.III - III1 × 3I such that I'T,.I(,) C T(,)I cverywhere. In this situation, if (x. u) ,ll {q} for SOlne q M2, then -(x) C I x {q}. Çonsequently, we bave AI l X {q} = --1(7(/l {q})). SO, th lçve [1 {q}, ('orrspolldllg to th prodllçt, project under to a foliation L on (M.g) and n-(L) = {Mt x {q},q Al2}. Let L be the distl'i]mtioli Oli Al tangent to L. I)cfinc the distribntion L2 to bc the orthogonal distrit)ution to Lt on I. Then where h denotes lhe horizontal lift. If X and ]" are vector fields on I tangent to L and if, i are tangent to L2, then X h, yh are tangent to 1 and rh, V h are tangcnt to I2. Because of the product strncture, we have that çx - and çg X arc t311gcllt to [1 lld Çg i h and Ç.x' Un are tangent to AI2. Using the expressions (1) for Ç, this means that Ç.x]" and ÇuX are sections of L" so, L is totally geodcsic and even totally parallcl: ÇUl" raid ÇA-U arc sections of L2: so, also L2 is totally geodesic and totally parallel (iii part icular, L is integrable with sociated foliation L2); R(U, V)u = R(X, U)u = 0: so, L2 is containcd il the nullitv distribution of the curvature. The leaves of L2 are therefore fiat. These properties imply that L and L.2 consist of the leaves of a local ielnamiian prodnct M M' x N where k = diln L n (sec [8]). Suppose converselv that I is locally isometric to M' x N for 1 " 5 n. This gives rise to two foliations on I: L = {M'x {v}. t, G N } and L2 = {{p} xN,p G I}. Define two COlnplelnentary distributions L and 2 on TI by L = l'Trl h(TM'). L = h(TNk). It is easily checked sing (1) that and {2 are totally geodesic and totally parallel complementary distributions. Hcnce, the leaves of their correspollding foliations È and are actually the leaves of a local Riemanniall product, hl particular, note that = {-(3I' x {t'}),v Na}. So. TM is indeed locally reducible. 6. (LOBAL RESULTS YVe continue with the notation of the previous seçtioll. In order to derive results concerlling the global reducibility of (TriiI, gs ), we will exploit the relationship between the foliations L1 and L2 of (il1, g) and the foliations t and 2 of (Tr]II. gs) iii t.he case of a vertical deconposition. "lVe ha.ve alreadv relnarked that L1 and 1 deternfine each or her reciprocally by L 1 = re (! 1 ) and = re- l L 1- The relat ionship between the foliations L2 and _ is hot so straightforward. We still have L = rc(Ï2), but deternfining L2 from L requires a little more tare. To construct the leaf ç of 2 through a point (x, v) T,.III, consider all the curves iii the leaf S of L_ starting at z III. Then, ç consists of all end-points of the horizontal lifts of these curves starting at (x, u). We call ç the horizontal lift, of S through (x, u). Since is evervwhere horizontal, the lnap re- -- S is a local isolnetrv and , is a Riemalmian covering of S. \¥hen S is silnply COlmected. 5} and S are globally isolnetric and. iii particular, one-to-one. 2902 E. BOECKX \Vith these comments in nfind, we now proceed to the proof of the Global The- orem. So, we suppose that dira M 3 and that (T,.M. gs) is isometric to a global Riemalmian product (kl, g) x {M2, g2). Since dira M 3, this is a vertical decon> position and I'T,.M is tangent to one of the fmtors, say M. Cnsider kl and as submanifolds of TAI {i.e., choose one leaf of eh of the foliations L and and define the submanifolds M' := (kl) and F := (M2) of M. From the local considerations in the previous section, we know that (M.g) is locally isometric to the Riemannian product M' x F and that F is fiat. We show that there is a onc-to-one correspondence between M and M' x F. Take a. point x kl and consider an arbitrary vector tt E Tkl of length r. Through (.. u) e Tal, there is a unique leaf = of L and a unique leaf 2 of L2- Because of the product structure on T,.M, cuts Al2 in a unique point O A& and cuts al, in a torique point al,. Put p := () ai' and q= := (0) claire that lhe con'espondcnce kl al' x F" ,," (v=, q) is well-defined, i.e., independent of the choice of the ta.ngent vector u. To ste this, take another vector ,, Tal ,f length r. Sin«e (, u) = a" = (.r. t,), the leaf of 1 contains both (.r, t,) and (,r, v); so we have u = ,'1, 0, = 0,. and q,, = q,,. The unique leaf of 2 through (a', t') is different from ç2.- However, both are horizontal lifts of S2, = w(2u). So, if #(t) = (.r(t),u(t)) is a curve in ç2u such that (0) = and (1) =,, ,/I, then 7 = ° runs flom,r tot M in $2,. Denote by the horizontal lift of 7 starting al (a',v). Çlearly, lies in 2 and ends On the other hand. starting from a couple (p. q) M' x F, we find the corre- sponding point ,r al as = u(, 0) for some al with () = p and the unique 0 M.e with (0) = q- Via an argmnent as above, one shows that does hot depend on the choice of and that the nmp (p, q) defined in this way is the inverse of the map (p, q). Next, we note that the correspondence kl M' x F: (p,q) is defined so as to respect the local product structure. In particular, the metric on corresponds to the product metric of M' x F, and the first statement is proved. Conversely, suppose that (M, ) is the global product space (M', ') x (F, g0)- By choosing a leaf of both product foliations, one can consider M' and F sub- manifolds of M. Let «0 be their intersection point and choose a vector uo ToM of length r. Define kl as the inverse ilnage of M' under the projection and k12 the horizontal lift of F through (a'0, u0). Since we suppose F to be simply connected, kl. is isometric to the fiat space (F. 0) and ,1 and I2 have {0. unique intersection point. We show that there is a one-to-one correspondence between TkI and kI x Take {, u) TM and denote by S1 the unique leaf of L 1 Oll al through m and by $2 the unique leaf of L2 on kl through Then, the leaf of through ( is given by -(S) and the leaf 2 of 2 through (, u) is the horizontal lift of through this point. cuts a& in a unique point O with (0) = S F, and 2 cuts kl in a unique point with () = $2 al'. (Note that the simply connectedness of F is essential to ensure uniqueness.) Çlearly, the correspondence Tkl 1 x M2" (, u) (, 0) is well-defined and it is hot dicult to construct it.s inverse. Since this correspondence also respects the local product structure, the metric gs on TM corresponds to the product metric on kI x k&. This completes the proof of the Global Theorem. WHEN ARE TANGENT SPHERE BUNDLES FIEDUCIBLE? 2903 Remark 4. The proof of the Global Theorem continues to hold when n = 2 for the case of a vertical global decomposition of (TAI, gs). Clearly, the base manifold is then fiat.. That we need the simply commctedness of the fiat factor tan be seen from the example of a two-dimensional fiat cone C The vertical and horizontal distributions on TrC are both integrable, and locally their integral manifolds are the leaves of the local product foliation on T,.C If it were a global decomposition, everv maxinml integral manifokt of the horizontal distribution would intersect ev- ery vertical fiber exactl.v once and it would be isometric to C under the natural projection 7r. This would define a global parallelization of C, contrary to the fact that its full holonomy group is non-trivial. IEFERENCES [1] M. F. Atiyah. R. t3ott and A. Shapiro, Clifford modules, Topology 3, Suppl. 1 (1964), 3-38. MR 29:5250 [2] J. Berndt, E. Boeckx, P. Nagy and L. Vanhecke, Geodesics on the unit tangent bundle, preprint, 2001. [3] J. 13erndt, F. Tricerri and L. Vanhecke, Generalized Heisenber 9 groups and Damek-Ricci harmonic spaces, Lecture Notes in Math. 1598, Springer-Verlag, Berlin, Heideiberg, New York, 1995. MR 97a:53068 [4] E. Boeckx and CI. Calvaruso, When is the unit tangent sphere bundle semi-symmetc?, preprint, 2002. [5] E. Boeckx and L. Vanhecke, Characteristic reftections on unit tangent sphere bundles, Hous- ton J. Math. 23 (1997), 427-448. MR 2000e:53052 [6] E. Boeckx and L. Vanhecke, Curvature homogeneous unit tangent sphere bundles, Publ. Math. Debrecen 53 (1998), 389-413. MR 2000d:53080 [7] E. 13oeckx and L. Vanhecke, Harmonic and minimal vector fields on tangent and unit tangent bundles, Differential Cieom. Appl. 13 (2000), 77-93. MR 2001f:53138 [8] CI. de Rham, Sur la reductibilité d'un espace de Riemann, Comment. Math. Helv. 26 (1952). 328-344. MR 14:584a [9] O. Kowalski and M. Sekizawa, On tangent sphere bundles wth small or large constant radius, Ann. Cilobal Anal. Geom. 18 (2000), 207-219. MR 2001i:53049 [10] O. Kowalski, M. Sekizawa and Z. Vl£ek, Can tangent sphere bundles over Riemannmn mani]olds bave strictly positive sectional curvature?, in: Cilobal Differential Cieometry: The Mathematical Legacy of Alfred Ciray (eds. M. Ferngndez, J. A. V'olf), Contemp. Math. 288, Amer. Math. Soc., Providence, RI, 2001. 110-118. MR 2002i:5304ï DEPARTMENT OF h[ATHEMATICS, KATHOLIEKE UNIVERSITEIT LEU\ EN. CELESTIJNENLAAN 200B. 3001 LEUVEN. BELGIUM E-mail address: cric. boeckx@wis, kuleuven, ac. ho TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2905-2923 S 0002-9947(03)03274-4 Article electronically published on March 17, 2003 CRITERIA FOR LARGE DEVIATIONS HENRI COMMAN ABSTRACT. V'e give the general variational form of lira sup(./i, e h( x ) / t tta ( dx ) ) t foi" any bounded above Borel measurable function h on a topological space X, where (tt) is a net of Borel probability measures on X, and (ta) a net in ]0. oe[ converging to 0. When X is normal, we obtain a criterion in order to have a limit in the above expression fol" ail h continuous bounded, and deduce new criteria of a large deviation principle with hOt necessarily tight rate functiom this allows us to remove t he tightness hypothesis in various classical theorems. 1. INTRODUCTION Let Cb(X) be the set of real-valued bounded continuous functions on a topological space X, (Va) a uet of Borel probability measures O11 .', and (/a) a net in ]0. converging to 0. For each [-o% +ec]-vahled Borel measurable function h on X. we write ptâ (eh/t) for (fx eh(x)/t° ta(d'r))t«" and define A(h) = log lira pta" (e h/t) provided the limit exists. The ailn of this paper is to clarify the relation between the existence of A(h) for all h Cb(X). and the one of a large deviation principle for (p) with powers (t). This problenl origillates fronl Varadhan's theorenl, which sta.tes that if X is regular. then such a prillciple with tigllt rate function .1 inlplies the existence of A(h) for all [-oc, +oc[-valued contilmous fimctions h on X satisfying some rail condition (in particular for ail h rb(X)), with moreover A(b) = sup:ex{h(x ) - J(:r)}. This theorem is a crucial argument in the proofs of all related results: in particular, these also hold under some tightness hypothesis (of the rate fimctiom or of the net (it2 i.e., exponential tightuess). We present here a new approach based on a variational representation of the fuuctional lim suppt2(e'/t), and a criteriou of existence of A(-) on Cb(X) wheu X is uormal. This leads to functional as well as set-theoretic large deviation cri- teria, which allow us to relnOVe the tightness condition in various basic results of the theory; moreover, the proofs are nonstandard since there are no compactness arguments in the entire paper. Notice that all the results work for general nets of measures and powers, and (except for Section 4 and "(i) (il)'" in Theoreln 3.3 where X is assumed to be normal) for a general topological states space. Received bv the editors January 3, 2002 and, iii revised form, November 9, 2002. 2000 Mathematics Subject Classification. Primary 60F10. This work was supported in part by FONDE('YT Grant 3010005. @2003 American Matheiuatical Society 2906 HENRI COMMAN We begin in Section 2 bv stating somewhat unusual equivalent definitions of a large deviation principle (Proposition 2.3) which imply the existence of a minimal rate flmction. In Section 3. we prove a general variational form of lim supp « (e h/t« 1y) for any Bord measurable flmction h on X and any Borel set }" C X sa.tisving the (local- ized) tail condition of Varadhan's theorem (Theorem 3.1). Bv the saine methods. we obtain a sucient (and necessary if X is normal) condition for the existence of A(h) for ail h Ch(X), in the spirit of the Port manteau theorem (Theorem a.a). A generalized version of Varadhan's theorem without anv tightness assumption is a direct consequence (Corollary 3.4). In Section 4, assmning that X is nornml, we look for necessary and sufficient con- ditions that A(-) (as a flmctional on Ch(X)) nmst satisfv in order to bave a large de- viation principle. This is obtained in Theorem 4.1. which gives rive such conditions: in particular, it states that large deviations occur if and only if lin A(hi) = A(h) for each increasing net (h) in C(X] converging pointwise to h C(X). As corollaries, va.rions basiç results hold verhatim without tightness ssumptions: this is the ce for the eqnivalence imtween the Laplace and large deviation principles (Crollary 4.2), and for thc variational form of a rate flmction (4.5): a large deviation princi- ple is characterized a convergence in a mrrow space of set-fllnctions much larger than capa.cities (Remark 4.5]: Crollary 4.3 improves Brvc's theorem by weakening the exponential tightness lgpothesis; (4.2) gives the infimum of a rate flnction J on anv closed set i,, terres of lira inf p" (-) (resp. lira sup p" (-)), generalizing a well-known expression of J. "2. PREL1MINARIES Let .T" (resp. Ç) denotes the set of closed (resp. open) subsets of X. For each O [0. +]-valued function f on X we denote by ] (resp. f) the least upper semi- continuous function on X greater than f (resp. the greatest lower semi-continuous function on X less tlmn f), and define a map 7I " 9 -- [0.+oo] by 7I(G) = sup.eG f(x) for ail G G {7. We collect here some characterizations of positive bounded upper senfi-contim- ous flmctions that we will use in the sequel. Lemma 2.1. Tbere is a bijection between the set of positive bounded upper semi- «ovtinuous fvnctios f on X and the set of maps ï " Ç - [0. +OE[ satisfying (9.1) 7(OGi ) =- sup',/(Ci) for all {Gi:i I} C g, given by the maps 7 tir(x) = infeç,=e 7(G) for all x X. f-*7EEf. Mor«ov«r, for ch positive bound«d fun«tion f on X. the followin properties hold: (i) f = fæ; (ii) f is the unique positive upper semi-continuous function h on \" satisfying (iii) sup: c f(x) = infG»:Gç Tf(G) for all C X: CIITEIIA FOl LAIGE DEVIATIONS 2907 Proof. Let f be a positive bounded upper semi-continuous flmction on X. For each 1'" C X we ha.ve supr f _< Sllp{jï<sup. f+e} f Sllpt f ÷ 6; since {f < supr f + e} is open and contains }', we obtain (2.2) sup f = inf GD;Gç whence f's = f" \Ve now prove that %% = "y for ail 7" ç --' [0, +oo[ satisfying (2.1); let 7 be such a map. , > 0. and dcfine G), = U {G E and for all x E G, we have f.c(x) <_ "y(G) <_ u so that [,J<a G,, C {f. < ,}. For ail aï {f. < ,} there is G 9 .c such that "y(G) < ,, and thus This shows that {f. < A} C QJ,«a G, and thus {tir < 3,} = QJ«a G, which is an open set, so that f. is upper semi-contilmous. Let G (with the convention supo = 0). If supa f < 3(G), then G C {f < for some e > 0. Since ri is upper Selni-continuons, {f < 7(G) -e} is open with {f < 3(G) -e} C G(a)_, and since 7 is clearly ilwreasing, we obtain () N 7({f < ()-«}) (G(a)-) N ()-«, which gives the contradictiou. Thus 7(G) = sup G f7 for all G Ç and the first assertion is proved. Let f be a positive bomided functioli on .¥. Then 7I is bomMed and satisfies (2.1), so that f, is upper semi-contimloUS with fT, f. For ail positive bomded upper semi-contimous fllnctions f f, we have I f, which implies f = f. This prove (i). If h is a positive upper Senli-Colitinuous flmction on X satis[ving % = 7I, then h is bounded and (il) follows ri'oto (i). Let Y C X. By (2.2) we have sup. f = infGDY.Geç suPG f. and since = I by (ii). (iii) holds. Let h = V{9 [0. +[x: % = YI}- It is easv to see that % = f, and since fh is upper semi-contimtous with fh h. we have Yh = fh = f whence h: fTh = f by (i). Thus (iv) holds. Definition 2.2. We say that (p) sa.tisfies a large deviation principle with powers (t) if there is a lower semi-continuous flmction .1 : X -- [0. :c] such that lira sup (F) _< sup e -d (x) <_ sup e- a (x) _< lira inf l,t« (G) xF xG for all F 9 r, G {ï with F C G. Them J is called a rate function for (#t«), which is said to be tight if it has colnpact level sets. Notice that in the literature, a large deviation principle is in general defined for 1/n nets (lt)e>0 or sequences t#, ,eN*- In the sequel, when we will refer to known results that will be proved again, we will hot make this distinction and state them with general nets of measures and powers. By (2.3), the following proposition shows that the set. of rate functions for bas a minimal element: it is the only Olm if X is regular since it is well known ([2], Lemma 4.1.4). Proposition 2.3. The following statements are equivalent: (i) (#,) satisfies a large d«viation princwle with pow«rs (t,). (ii) There is a map 7 : ç--' [0, 1] such that (a) limsup#t°(F) N /(G) N liminfpto(G) for all F .. G çwth FcG. (b) "y(Uiel ai) = supiel "/(ai) for all {ai : i I} C ç 2q08 HENRI COMMAN (iii) There s et fiction f : .V -- [0, 1] such that limsupptâ(F) _< sup f(:r) _< liminfltâ (G) for «dl F G.. G E Ç u,ith F c G. (iv) There is a. fu.7,ction f: X [0.1] stch that limsul,/,tâ (F) _< SUl, f(.r) _< lira inf p°(G) Jbr a, ll F G., G G ç with F c G. I.f (i) hot&. then (i) holds wth r«te fmctio .1 given (2.3) e-J= V{ f G [0.1] "x "f s«tisfi«s (iii) (resp. (iv))}. altd (2.4) 7¢-a = V{G [0.1] ç "7 satisfies (ii)}. q (il) holds u,ith , thon (i) hold.s" with rate f«nction - log G where J'v(a') = inf 7(G) for ail z A. (f (iii) (rep. (iv)) bol& with f. thett (i) bol& u,ith rate fu,ctio -logf. Proof. If (il) holds with 7- then f. is upper semi-continuous and (i) holds with rate fim«tion - log f. by Lemma 2.1. If (iii) holds with f, then (il) holds with 7, and so (i) holds with rate function - log , since fw = f by Lemma 2.1. If (iv) holds with f, then pu 7(G) = suprcG supr f for ail G G ç. and notice that 7 satisfies (il). Thus (i) holds with rate fimction -logfT. Since f we hRvo sup f sup f sup f = sup f F F G G G for ail F U, G G Ç with F C G (the equality follows from Lemma 2.1 (iii)). Thus (i) holds with ra.te fimcion - log f. If (i) holds, then (il), (iii) and (it,) hold. The fimçtion t, = {f [0.1] x -f satisfies (iii) (resp. (it,))} obviouslv satisfies (iii) (resp. (it,)); the saine for h bv the preceding discussion, and h = by the definition of h; put e -J = and obtain (2.3). The map [0, 1] 9 7 satisfies (ii)} satisfies (il), and so (i) holds with rate funct.ion J given bv c-J = fTt. Since 7-a = 7t = 7t, (2.4) holds. Corollary 2.4. (Contractio principle) Let Y be t a coti.uous fu.ctio.. If (p) satisfies a large deviation priciple with powers (G) ad rate fueto, jx. the. ([p]) satisfies a large deviatio, pri.ciple with 0 powers (G) and rate functio, jv = 1 where l(v) Proof. Let jx be a rate fimction for (it ). The relations i.n .p [,d (c) = in p G (- I(F)) _j _d x sup e sup « liminf[p]t(G) -(F) -(G) CR1TERIA FOR LARGE DEVIATIONS 2909 for all F E .', G E çwith F C G and Proposition 2.3 show that (Tr[tt,]) satisfies a large deviation principle with powers (t) and rate functiol JY = -log] where jx jx f(/) = Sllprr-,(y ) e- for all g G Y (Sillce Sll])r-l(F ) e- = SllPF f). Equivalently, jr = l where l(g) = illfr-l(y ) .J\" for ail g G I'. [] 3. A GENERAL VARIATIONAL FORMULA Up t,o now, the onlv known condition that ensures the existence of A(h) for all h Ch(X) (and lnore generally for all [-c, +oe[-valued COllt.inllolls fllnctions ]2 on X satisfving the tail condition (3.1) lira limstlp/tW (e h/t l{eh>^t} ) = 0) is the existence of a large deviation principle with tight rate fimction, and A(h) is expressed in a variational form in terres of this rate fimction (Varadhan's theoreln, [2] Theorem 4.3.1). In this section, we generalize these results in two directions. First, Theoreln 3.1 gives the general variational form of lilnsllPltâ ' (e h/t° lr) for any Bord set }" C X and anv Borel measurable flln('tion ]/ on ." satisf.ving the localized tail condition (3.2). Next, Theorem a.a gives a sufficient condition for the existence of A(-) on Ch(X) which is also necessarv when X is nornlal; moreover, the variational form of A(-) is obtained in terres of any set-function 7 6 [0.1] satisfying the typical in-between inequalities of large deviations (a.ua). As a COl- sequence, Varadhan's theorem is gelmralized in various ways (Corollary 3.2 and Corollary 3.4). For each nlap h X + [-oo, q-oo] we put F h - {e h E [ - 6, q- 6]} and G h = {e h ]A - e, A + e[} for ail A > 0 and for all e > 0. Theorem a.1. For each Bord set t" C X. and for each [-0% +oo]-val.ued Bord measurable function h on X satisfgin9 lira liln sup/zta (e h/t° l{>^t}c)') = 0. hI--ec (3.2) we bave (3.3) (3.4) for some 11 e [0, +ce[. Moreover. li,nl, t(e h/t° 1,-) ezists if sup {(A-g) limilfpâ°(FÂ, efY)}= sup {(-e) lilnsupHta(Fîeç}')}. A_>0,e>0 A>_0,e>0 Pro@ Let Y be any Bord subset of X. Put 9 = e, G.e = Gî,«Y, F.« = Fî.eY for ail A > 0 and for all « > 0. have limsup«(91/tly) limstlp(91/tlF.) fo If x ( - e) lira sup g [ ,«) for all OE 0 and for all e > 0, and so (3.5) limsupg(g/tly) sup {(-e) limsup(F,«)}. 2910 HENRI COMMAN (3.7) Since Thus, in oder to prove (3.3) and (3.4), we have to prove that for SOlne/[ < oe, (3.6) lilns,,PPâ (gl/t«l,) i sup {x;«>o:g(x)_<M} For ail M 0, for ail N N* and for ail 1 G j G N, we define Iç hae N 1/t j=l llI3X lims,,p l, (glt 1FM..,) V liIllSllp « (gl/G l{g>M}y)" lira slip p (1/t ] FM,N,J ) (__ |illl 8/lp ]lta c' (&[,N,j)llg F,.. II, it follows from (3.7) that lira sup , (9 /t 1 ) (3.8) max ]]91FM.]llimsuplz(F,,N.y) vlimsupl*(91/tl{e>t}m'). IjN ' Let AI , N in (3.8) and use (3.2) to obtain solne Alo [0, [ such that (3.9) lilnsuplt«(gl/t«ly) liminf lnax{I]glFM.m, lllilnsupp (F,NO)}. Thus, to obtain (3.6) it suffices to show lilll inf lnax { [91F,.. [1 linl 8up , (Mo,N.j) } (3.10) 8Up {(g(x)--)liln8tlp,oE(g(x),¢)}. If (3.10) does not hold, then there exists oe > 0 such that lira inf lnax { I[gl F»,,, H liln slip ,â (Mo,N,j) } Noe N- (3.11) > sup {(g(x) + p E) lilll sup t« - , Take 0 < o < oe/2 in (3.11) and obtain linl inf max {[]glFMo.m l linl slip Ira « (FMo.Nd) } (3.12) > sup {(g(x) + «o) lira sup " (G(.),«)}. But for ail 0 A Mo and for ail N > Mo/«o we hae (3.13) (A + Eo) liInsup t l t'F , (a.)> lbu,., llil,p. t ,..) where jx is such that k e [(jx-1)ao/N, jxMo/N] (since [(jx-)Mo/', ix alo/N] c ]A - «o, k + «oD- When A ranges over [0, klo], jx ranges over {j" 1 j N}, and (3.13) implies O<<Mo (3.4) 2 max {llg,,..[limsup(F, NO)} CRITERIA FOR LARGE DEVIATIONS 2911 for ail N > Mo/eo. Notice that ri)r ail N E N* a.nd for ail 1 _< j _< N, if then j = Jg() for some x Y. Thus it suffices to consider A {g(x) Mo} in the L.H.S. of (3.14), that is, snp {(g(x) + eo)lin, supltL«(G9(),eo)} for all N > Mo/eo, which contradicts (3.12); it follows that (3.10), (3.6), and finally (3.3) and (3.4) hold. In the samc way that we obtainod (3.5), we have (3.15) liminf#L«(gi/t«ly) su I) k0,e>0 and the last assertion follows from (3.4). A localized version of Varadhan's theorem states that if X is regnlar an(l if satisfies a large deviation principle with t)owcrs (te,) and tight rate function .]. then (3.18) and (3.19) hold with 1 = J ([2], Exercise 4.3.11). Thc following corollary removes all the hypotheses on / and X. Corollary 3.2. Let 1 be a [0, +oe]-valued function on X satisfying (3.16) F , limsup#L«(F) supe -() (3.7) (re.»p. G Ç, liIninfpL«(G) k suI) e-()). xG Tben, for each [-oe, +oe]-valu«d continaous function h on X satisfying (3.1), we (3.18) F , liInsupltL«(«n/t"lF) 5 sup eh()« (3.19) (r«sp. G Ç, lira inf ltL « («h/t 1) k sup «h()«-()). Proof. Suppose that (3.16) holds and (3.18) does hot hold for some [-, valued continuous function on X satisfying (3.1). Since for all F G U by Theorem 3.1, there exists Fo G U, xo G Fo with h(xo) < OE, a.nd eo > 0 such that (e (°) eo)limsup/tL" F sup eh(x)e -l(x). xEF, h(x)< By (3.16) we have (e h(*°) - 0) sup e -l(*) > xE Fh( o) .o nF and so there exists x Fh(o),o ç) F such that (e h(x°) -- ¢Eo)e -t() > sup eh(x)e-l(x. Since e h(*) _> e h(*°) - eo we obtain sup eh(*)e -t(x), xF,h(x)<oe xF,h(x)< c eh(**)e-l(**) > sup eh(*) e -t(x) 2912 HENRI COMMAN with xl Ç F and h(Xl) < , whence the contradiction. Supt)ose now that (3.17) holds and (3.19) does hot hold for some [-, +]-valued (.oltillUOUS ftulctioi1 h on X satisf'ing (3.1). By (3.15), there exists C0 Ç Ç such that sup e()e -/() > lira inf « (e /t« 1o) xEGo,h(x)<oe sup {(e h(z) - ) lira inf p.â (G he(« Go)}. {x6Go Thus. there exists .vo Ç Go with h(xo) < , and u > {} such that ch(*°)e -t(z°) > u + sup {(e h(x) - ¢) lira inf p. «"¢),e Go)} {xGo,e>O:h(x)<oe} and 1)y (3.17) «t(e°)e-I(x°) > e t(x°) sup e -l(x), x- Gêh(xO) ,eO CIG which gives the contradiction. A direct consequence of Corollary 3.2 is that Varadhan's theorem can be stated verbatim for a general state space and with any function (in place of a tight rate flmction) I X [0. +] satisfying the large deviations lower and upper bounds: (3.21) limsupp,(F) supe -() 5 supe -() liminflt(G) xGF xGG for ail F Ç . G G Ç wi/h F C G: that is to say. A(h) exists and ,X() = ,,p {() - («)} xX,h(x)<oe for ail [-OE,, +.]-valued continuous flmctions h on X satis6"ing (3.1). X will see wi/h Corollary 3.4 that it is possible to go fur/her in the generalization of Varadhan's theorem ot)taining/he saille coilclusiOlS with hypothesis weaker than (3.21). ecall that X is norlnal if and only if the following interpolation property holds: if f and g are real-valued respectively upper and lower seini-coiltinuous functions on .k" such tha/f g, then there is a continuous fllllCtiOll Oll X satis6"ing f 5 h g. Theorem 3.3. Consider the followig statements: (i) A(h) exists for all h Ç Cb(X); (il) limsuppâ(F) liminf p.â(G) for all F G . G G Ç with F C G. If X is normal. then (i) (il). If (ii) holds, then (i) holds and moreover for each [-, +]-valued cont.inuous Iunct.ion on X satisfying (a.1) we hat, e Ior some ai [0, +[. (3.22) e h(h) = slip {()-e)7(Fî, e) } =- A>_0,e>0 for all maps 7 " .7 tO ç -- [0.1] satisfging (3.23) {xX,e>O:eh() <_M} limsuppto(F) <_ -)(F) <_ 7(G) <_ lira inf CRITERIA FOR LARGE DEVIATIONS 2913 for all F E., G E ç with F c G. Pro@ Suppose that (i) holds and X is normal. For each F ff and G ff Ç with F C G, there exists h Cb(.') such that 1F h 1 G. Since 1F e nh-n e-"l.x.kG V 1G for all N, we obtain lira sup p (F) 5 inf e A[nh-n) inf liminf{e - + #tg(G)} liminf p,(G) and (il) holds. Suppose that (il) holds. Let h be a [-, +]-valued continuous fimction on X satisfying (3.1), and 7 " lU Ç + [0, 1] satis(ying (3.23). Put 9 = eh and let us use the saine notation as in the proof of Theorem 3.1 (with l = A-). For all A 0. for all e > 0 and for all 6 > 0 with > e, we have by (3.23), lira inf item(9 /«) lira inf 1,(9 /t (- 6)7(G,6) (A- 6)7(F,«). Thus and lira iaf p.to (91/to) >_ lira (,k - 5)3'(FA,e) _> (,x- e)7(F,) _> ()- liminf p, tâ(91/t') >_ sup A>_O,e>O > ,,p A>O,e>O In order to prove (3.22), we bave to prove that for some M < (3.24) limsupp,(g 1Ce) sup {(e h() - «)(G,«)}. { x X,e >O:eh( I } Bv using (3.23), and in the saine way that we have obtained (3.10) in the proof of Threm 3.1, we find some M0 [0. [ such that to prove (3.24) it suffices to prove which is achieved exactly as in Theorem 3.1. The following corollary gives sufficient conditions much weaker than large de- viations with tight rate function in order to have the conclusions of Varadlmn's theorem; in fact, we will see in the next section (Corollary 4.2) that when X is normal, the condition (3.25) is also necessary. Corollary 3.4. Let 1 be a [0, +oo]-valued function on X satisfying (3.25) lira sup p,to (F) < sup e -t(x) < lira inf lâ ° (G) x6G (3.26) (resp. lira sup #tâ (F) for alI F . G E ç with F c G. Then, A(h) ezists and (3.27) A(h) = sup {(x) - xX.h(z)<oc 2914 HENRI COMMAN for all [-oc, +oc]-valued continuous functio. h on X satisfyi. (3.1). Pro@ Let h be a [-oc, +oc]-valued COl,tilmous fullction Oll X satisf.ving (3.1). If (3.25) holds, thon by Theoreln 3.3 (with 7(G) = supzec; e -t(z) for all G E Ç), A(h) exists and e A(h) ---- sup {(A -- e) sup e-l(z)}. Since for all A _> 0, e > 0 and x E Gî., (A -- e)e -l(z) <_ eh(z)e -l(z), we obtain and thns C A(h) _ Sllp eh(z)e -l(x). xEX,h(x)<oc For all x X with h(x) < oc, and for all e > 0 we have (e h(x) -- ¢)e -l(z) <_ (e h(z) -- 6) sup YGêh() ,« which implies (eh(z) _ )e-l(:r) <_ e A(h), eh(z)e-l(z) <_ e A(h), and finally sup eh(z) e -(z) e ri(h). xX.h(x)<oc Thus e h) = supex.h(z)< eh(X)e -t(z), which is equivalent to (3.27). If (3.26) holds, we conclude by applying Theorem 3.3 (with 3'(F) = supzeF e -(z) for ail F 9r), and replacing G h,.« bv. Fî.«, and G heh(), e bv. Fëh(), e in the above proof. [] Remark 3.5. Let F be the set of lnaps - : 9 r U Ç - [0.1] such that r(F) _< r(G) for all F 9 r, G Ç with F C G. Define the narrow topology on F as the coarsest topology for which the lnappiugs 3' * 3'(}') are upper semi-contilmous for all Y E 9 r and lower semi-continuous for all Y ç The net (#" (.1/t,)) can be seen as a net in F provided with the narrow topology, as well as a net iii [0, provided with the product topolog:y: Then. the implication (ii) => (i) in Theorem 3.3 means that if (tttâ(.1/t«)) has a limit iii F, then (tttâ (-1/% )) has a limit in [0, oc[{h:heG(¥)}; lnoreover, the converse holds if X is normal. Of course, the limit in F when it exists is hot Ulfique: for each F 9 c and G ç " defined bv -(F) = lira sup #tâ (F) and "(G) = lira inf ptâ (G) is an example, and "' defined bv ")"(G) = 7(G) and 3/(F) = infa»F.aeç 7(G) is another oue. CRITERIA FOR LARGE DEVIATIONS 2915 4. CR1TER1A OF A LARGE DEV1AT1ON PR1NCIPLE In this section, we investigate what has to t)e added to the existence of A(h) for ail h E Ch(X) (in other words, of the limit A(-) of (logltt«(e /t° ))iii ]-o% +oc[ ch(X)) in order to have large deviations. Of course, SOnle hyl)otheses on X are required to have sufficieatly continuous functions; so we sui)pose here that X as normal. In this case, by Theorenl 3.3 (alad Relnark 3.5) the existence of A(-) on Ch(X) as equiva- lent to the existence of a narrow set-theoretic lilaait 7 E F of (pt«), which as also equivalent to tlae existence of A(h) for ail [-oc, +oc]-vahled continuous finl«tions h on X satisfying tlae rail condition (3.1); nloreover, fi)r eacla such finlction h. the variational form of A(h) as given ila terlns of '7. In I)articulal, "7 can vary and it as essentially this flexibilitv which allows us to o|)taila in Theoren 4.1 lmcessary and suflïcient conditions, each of thenl corresponding to sonle tyi)e of infornmtion: a property of A(-) as a fun«tional iii (ai), a special variational fol'ail of A(') in (iii), a property of "7 iii (iV), and a property of the net (pt) ila (v) and (ri). It as worth noticing that in hoth formulations (flmctional (il) or sct-theoretic (iv)), the condition on the lilnit as thc saine: a continuitv property on ilacreasing nets. As corollaries, several basic results of the thcorv are strelagthened bv realoving the tightness or compactncss tayi)othesis. Theore,n 4.1. If X as no,wtal, then the follou,ing statements are equivalent: (i) (p) satisfies a large deviation principle wilh powers (t,). (ai) A(h) exisls for all h Cb(X), and A(b) converges fo A(h) for each increa.s- in9 net (ha) in Ch(X) converging poinlu,ise lo h Cb(X). (iii) A(h) exisls for all h Cb(X), and A(h) = supex{h(x ) -/(.r)} for some function ! : X [0. +oc] and for ail h Cb(X). (iv) There as a map "7 : Ç -- [0.1] such that (a) lilnsuplt(F) _< "7(G) < linlinfpt(G) for all F ., G ç wzth FcG. (b) "7([.J, Gi) = linl'7(G) for each increasing net (Gi) i Ç. (v) A(h) exists for all h C(X), ad for all F .T, for all open covers {Gi : i I} of F and for all e > O. there exists a finte subset {Gq, .... G u } C {Gi : i I} such that (4.1) lilninfpt(/)- limsuppt( U Gi) < ¢. (va) There as a function l : X -- [0, +oe] such that (4.2) inf/(x)= sup {-liminftclogtz(G)} = su v {-limsuptclogp(G)} xF GÇ,GDF GÇ,GDF for all F .. If (i) holds with rate function J. then the following properties hold: (4.3) inf J(x)= sup {-A(h)} for all F ,'; xGF h.Cb(X),hlF=O (4.4) inf J(x) = sup {-A(h)} for ail G ç, xG hCb ( X ),eh _lG 2916 HENRI COMMAN where Che(X) is the set of [-oc, +oe[-valued bounded above cotiuous flmctions on X ; in particular, (4.5) J(x) = up {h(x)- A(h)} I« ll x e X- o (4.6) J = 1 yor ,,Il l" X -. [0. +] sti.yu,,9 (iii): (-1.7") e- J ( x ) = inf GEç,xEG ]br all x e A. md for all 3" Ç [0.1] sotisfying (iv): (4.8) ,l = 1 for all l" X [0. +oe] satisfying (t,i). If morcover X is second cou,table, thon we tan replace "'net" bg "sequence" in (ii) Pro@ (i) (iv) and (iii) (ii) are «lear: (i) (iii) bv Corollary 3.4 and so . V (i) (ii) If (i) holds with rate flmction J. then for each F G . each open co er {Gi " i G I } of F and each e > 0, lira sul)l,â ° (F) < sup e -J < sul) e -J = sup sup e -J < sup lira inf pt« (Gi) ÷ 6, which implies (v). Suppose that (il) holds. We will prove that (i) holds. Let Che(X) be the set of I-et, +ec[-valued bounded al)ove continuous fimctions on X. By Theorem 3.3, A(h) exists in [-OE, +vc[ for ail h Coe(X), and notice that A(h V k) = A(h) V A(k) for all k Che(X); in particular. A(h V s) = A(h) V s for all s e [--ec,+ec[. Let (bi) be an increasing net in Che(X) converging to h Che(X) with A(h) > -ec. For each real s < A(h) we have limA(hi V s) = A(h V s) = A(h), and so eventually A(h) > s, which shows that lira A(h) = A(h). Therefore, we can replace Ch(X) 1)y Che(X) in (ii). Let F e .T and h Che(X) with hlF = 0. If A(h) > -oe, then A(hVs) = A(h) with (hVs)l F = 0 for ail s < A(h) A 0: if A(h) = -oe, then the sequence (A(h V --*))eN converges to --OE with (h V -n)l F = 0. Thus, inf e A(h) = inf e A(h), h Cba ( X ),h l oe=O h Cb ( X ),h l F =O and bv the interpolation property we have lira sup tt2 « (F) <_ inf e h(h) = inf e ri(h) h (7. Cba ( X ),h l F=O h (7.Cb ( X ),hl F =O (4.9) _< sup e h(h) <_ lira inf p (G) hGCba(X),eh_lG for ail F G .T, G G _G with F C G'. Define f() = inf c A(h) (= inf e't(h)) hGCb(A ),h(x)=O hGCb«(X),h(x)=O CRITEIRIA FOIR LAIRGE DEVIATIONS 2917 for all x C A'. By (4.9), in order to prove (i) it sulïices to show that f is upper semi-continuous a.nd sa.tisfies (4.10) ,qU I) f(x) = inf e A(h) xE F hECb ( X ).h I oe =0 for ail F C - and (4.11) su I) f(x) = sup e A(h) x:G hCb« ( X ),eh _lG for all G Ç. We first show (4.10). Ch'arlv sui) f(x) <_ inf e A(h) xÇ F hCb( X),hlF=O for all F -. SuI)I)ose that sup f(x) < e s < inf e A(h) hGCb( X ),hlF=O for some F - and some real s. Then, for all .r F there exists h« C Cb(X) «hich tan bc chosen negative such that ho(x) = 0 and (4.12) A(h«) < s < inf A(h). hECb(,\ ),hlF=0 But 1F <_ e V¢ h with VeF h« bounded lower semi-continuous, and so there exists h Ch(X) such that 1F N e h e Ve h (in particular hIF = 0). Let I be the set of finite subsets of F ordered by inclusion, and bi = h A Vzei h« for all i I, so that (bi)iCI is an increasing net in C(X) converging to h. Since A(hi) A(Ve h) = supzei A(hz) < s for ail i I. we obtain limA(hi) = A(h) s, which contradicts (4.12). Thus (4.10) holds. now prove (4.11). Bv the interpolation property (between 1{«} and 1GE) ve have clearly sup f(x) sup e A(h) xÇG hCba( X).ehlG for ail G {ï. Suppose sup f(x) < sup e A(h) X _G h_Ctm (X) ,e h 1 G for some G {ï. Then, for ail x G there exists ha Cb(X) with ha(x) = 0 such that (4.13) sup A(h) < s < A(ha) xG for some ha Cba(X) with e ha <_ 1, and some real s. Let I be the set of filfite subsets of G ordered by inclusion, and bi = b A /ei hx for all i I, so that (h)ie, is an increasing net in Cba(X) converging to hE. Then A(h) = limA(hi) <_ limA( V h)= lim(supA(h)) <_ s, xi xEi which contradicts (4.13). Thus (4.11) holds. It remains to show that f is upper semi-continuous. By (4.9), (4.10), (4.11), and since f(x) = inf e () _ < ilf sup e A(h) hÇCba(X),h(x)=O GD{x} hCb«(X),e <_IG 2918 HENRI COMMAN for ail x E X. by Lennna 2.1 il suflïces to prove that (4.14) f(x) = illf e A(h) >_ |nf slip e A(h) hECba (X),h(e)=0 GD {e} hECba (X),e h for all :r E N. Suppose that (4.14) does not |iold for solne :r X. Then. there exists h« Cb,(X) wit.h hx(x) = 0, and oe > 0 such that (4.15) eA(h) + u < |nf sut) GD{x} hCb«(X},eh_lci I3y (4.9) and Theoreni 3.3 (with 7(G) = SllPheCba(.\-),eh<_lci eh(h)), we have (4.16) e A(h) = sui) {(A - ¢) slip eh(h)}. .X>O,e >O h_ Cba ( X ),e h (1 Take A = 1 alid 0 < e < ,, in (4.16), and obtain by (4.15), mil» c A(h) < |nf sui» hCba ( X ),e h (1Gh x GD { e } hECba ( X ),e h 1 ci h, wit|l x Gi., wh|ch gives the contradiction. Thus (4.14) holds and f is tlpper Selni-cont illllOllS. We have proved (i) = (ii), and that when (i) holds with rate flinction J, then (4.3) and (4.4) hold (by the miiqlleness of a rate flnlction on regular spaces); since A( - ()) = ,(,.) - (). (4.5) follows froln (4.3). Suppose that (iii) holds with l" X --, [0, +oc]. Then, obviouslv (il) and so (i) hold; let J be the associated rate filllctioll. By Corollary 3.4, (4.17") e A(h) --- slip ehe -J for ail h Cb(X). and so (4.18) X Slip h--J = sup h-I X X for ail h Ch(X). Clearly, for each h Cb(X) there exists a real s such that Slip «he--J = sup ffhVse-J and and so by (4.18), (4.19) X X slip eh -l --- Sllp ¢hVse-l, X X SUp c e- J = Slip eh e -I. X X For any G Ç choose an increasing net (h) in Cb(X) such that slip/e h` = 1, and obtain by (4.19), slip e - J = Slip e -I. G G o o Since supG e -I - supG e -I by Lemnla 2.1, we llave supGe -d = supGe -I for ail o O G Ç. Since e -J and e - are upper selni-continuous, we have J = 1 by Lenmla O 2.1. Thus, if (i) holds with rate fUllctiol ,J, then J = 1 for all 1 X satisf,ving (iii) and (4.6) holds. CRITERIA FOR LARGE DEVIATIONS 2919 Suppose that (iv) holds with 7 " {7 --, [0, 11. Define 7(F) = infa»F 7(G) for ail F E f', and notice that 7 is increasing on f', sat.isfies 7(F) <_ 7(G) for all F E f', G Ç with F C G, and by (a), (4.20) -( [_J a)<_ S,lp-r(aj) I<j<_N I<_j<_N for each finite family {Gj}I<j<N h _ Che(X) and (4.21) e A(h) =- (4.22) C ç Bv Theorem (3.3), A(h) exists for all We will show t.hat (il) holds. Let. (h)e bc an increasing net in Ch(X) converging to h Ch(X), and suppose that A(h) > supe A(h). By (4.21) and (4.22), there exists h0 > 0 and e0 > 0 such that (Ao - eo)7(G,« o) > sui) sui) {(£ - e)7(Fî,) } iI iI and thus h 7EE(Gîo,« o) > sup 7EE(Fî,e o) > sup'y(GAo,eo). iI iEI Let » be the set of finite subsets of I ordered by inclusion, and obtain by (4.20), h (4.23) Vil e , ?(Co,«o) > sup?( ' G,«). But G h h ao,«o C sup¢e Uie¢G)o,eo, and the condition (b) contradicts (4.23). It follows that A(h) = supie/A(bi), that is, (ii) and so (i) hold: let J be the associated rate function. We now prove that (4.7) holds. Let G ç and h Che(X) be such that e h _< 1«. For all x X and e > 0 with e h(x) > e, we bave and Thus, (4.24) sup Fêh(),« C G (eh(x) _ {(eh(:r)--g)'7(Fêh,=),e) } <_ sup {xX :,e h(=) >e},e>0 <_ and since if e A(h) > O, (4.25) e A(h) = p {(eh() _ {xX;e h(x)>},>O by (4.24) and (4.25) we obtain (4.26) sup hCba(X),ehlG e ri(h) _< 7(G). 2920 HENRI COMMAN Sul)pose t hat sup e A(h) + ve < hECba( X ),e h lG for some u > 0. Bv taking A = 1 and 0 < e0 < u/2 in (4.22) we obtain (4.27) ?(Feo) + u/2 < ff(G) for all h Cb.(X) such that e h 1G. Let (hi) be an increasing net in Cb.(X) such that sut) / e'h' = 1G, and let p be the set of finite subsets of I ordered 1)v inclusion bi Then (ieZ G,eo)e is an increasing net in Ç such that By (4.27) we bave and by (4.20), ri e ,, .,/(c,î:«o) + ./2 < ,,:(#* _ ,,«o) + ,/'2. < .,/(a), and by (4.4) sup e -a = -(G), G which gives (4.7) by upper senfi-continuity of e -d. Suppose that {v) holds. Bv Theorem 3.3. c A(h) .... sup {(A ¢)liminfpL"(Gî,e)} sui, {(A e) limsupp>(F,e) }h A0,e>0 AO,e>O for all h Cb(X). Let (h)e« be an increasing net in Cb(X) converging to h Cb(X). will prove that limA(hi) = A(h). If A(h) > supie A(hi), then there exists Ao 0, o > 0 and oe > 0 such that (4.) (Ao eo)liminf h - (G,«o) > sup sup {(A-e)limsupp(Fî')}+u iÇI AO,e>O iEI AO,e>O iI Take eo < e < eo + u/2 in (4.29) and obtain (4.30) (Ao eo) liminf Xo,«o/ > (Ao - eo)suplimsupt'Gh't o,«)" + /2. i6I Put F = F h Gi xo,eo' = Gxo,« for all i I, and notice that F C ietGi" Since O FD G h xo.eo we obtain by (4.0), O lira inf iI CRITERIA FOR LARGE DEVIATIONS 2921 alld so liminfPâ(l) > lilnsut'#â( U Gij)+ t/2, for ail finite subsets {G., " 1 j N} C {G " i e I}, which contradicts (4.1). Thus limA(hd = A(5), tiret is, (il) and so (i) hold. It remains to prove (i) (ri), (4.8) and the last assertion. Supl)ose that (i) holds with rate function .1. Put f = e -a and let F Y. By (4.3) we have sup f(z) = inf e A(h) inf lintinfp'(G) inf limsut)l,(G). x F hCb ( X ),hI F=O Gç,(IDF GÇ,GD F Suppose that inf e A(h) < inf liln sup tt hCb( X ),hlF=O GÇ,GDF Then. there exists > 0 and hF Ch(X with hl F = 0 su«h that e A(h) + < inf liln sup p GÇ,GDF (4.31) Since (4.32) e A(hF) sup { (c hE(z) - by Theorem 3.3. by taking e = e0 < iii (4.32) we obtain by (4.31), s t(F(, eo)} ) < inf limsupl,k (G). (4.33) sup {e h(x) liln up Pa { x X ;h F( x) < } , GÇ.GD F h Since hF(X) = 0 for all G F, we have F C G.eo liln sup p ,hu < lim sup p. ,,« _ (Ft,«) < hF C F[,«o and (4.33) ilnplies inf lira sup pâ (G) GÇ,GDF By combilfing Theorem 4.1 with Corollary 3.4 and Lemma 2.1, we obtain in the following corollary necessary and suflïcient conditions in order that a large deviation o principle occurs with rate function the lower regularization 1 of a given flmction l" X --, [0, +oo]. Notice that by Proposition 2.3 and (i) ¢ (ii) in Çorollary 4.2. the infilnum of the set of [0, +oo]-valued functions I on X satisfying (3.25) coincides o with t.he lower regularization l of each its elements. The equivalence (i) = (iii) in and the contradiction. We have shown that sup f(x) = inf lira inf pta° (G) = inf lira sup pta (G) for all F xF GÇ,GDF GÇ,GDF which is equivalent to (4.2) with l = J. and so (ri) holds. If (ri) holds with l-X -- [0, +cci, then (4.2) ilnplies limsup #C(F) <_ supe - <_ liminf pâ° (G) F for all F .T, G Ç with F C G. By Proposition 2.3. (i) holds with rate ftlllCtiOll o l and (4.8) holds. If moreover X is second countable, then it is well knowll that for any familv {bi i I} of lower selni-continuous functions on X there exists a countable subset I0 C I such that supie bi = supie o bi. It is easy to sec iii the above proof that this property allows us to replace "net" by "sequence" in (il) (resp. (iv)), and "open covers" by "countable open covers" in (v). [] 2922 HENRI ÇOMMAN Corollary 4.2 was known when 1 is a tight rate flmction ([2], Theorem 4.4.13): here there is no hypothesis on !. Corollary 4.2. Suppose that X is normal, and let I be a [0, + ]-valued function on X. Then. the following statements are equivalent: (i) (#a) satisfies a large deviation principle with powers (fa) and rate function o l. (ii) limsuppL " (F) _< sup e -(*) < lira lllft, a ((7) xEG for all F , G çwith F c G. (iii) A(b) exists and A(/,) = stlp{/,(x)- l(a')} for ail l, C Cb(X). xEX (iv) A(h) exists and A() = ,,p {h(«)-()} xX,h(x)< or all [-, +oe]-vol,,ed «o,tin,,ous unçtios o X sati&ig (3.1). Pro«¢ (ii) (iv) (iii) by Corollary 3.4. (iii) (i) by Theorem 4.1, and (i) (ii) since supxeG e -(*) = sup,eG e -(*) for ail G Ç by Lemnm 2.1. Recall that (Po) is said to be exponentially tight with respect to {ta) if for all g > 0 there is a compact set h C X such that lim supp (XkK) < «. Brvc's theorem ([2], Theorem 4.4.2) stores that if A(b) exists for all Cb(X) and if (p) is exponentially tight with respect, to (t), then {p) satisfies a large deviation principle with powers (); moreover, the (necessarily tight) rate function satisfies (4.5). The following Corollary 4.a shows that the first conclusion holds under a hypothesis clearly weaker thon exponential tightness. Moreover, Theorem 4.1 stores that without anv tightness hypothesis, a rate function for (pâ) always satisfies (4.5). Corollary 4.3. Suppose t5at X is normal. If A(h) ests for all h Ch(X), and gE for all open covers {Gi i I} of X. for all e > O. there exists a finite subset {«,, .... « } ç {«. e } « a (4.34) limsupl,(Xk Gi,) < g, t5e (I') satisfies a large deviation pri,,çiple with powers (). Pro4 Let F , {G i I} be an open cover of F and e > 0. Then. 0e G O XkF is an open cover of X, and so there exists a finite subset {Çh,----Çi } C {Çi " i I} such that linlsup#â(Xk( Gi, UXk )) < g. IjN Thu8, lira sup #t (F \ U and bv Theorem 4.1, the conclusion holds. CRITERIA FOR LARGE I)EVIATIONS 2923 Corollary 4.4. If X is 'normal and (lttâ ) satisfies (4.34), then (#,) bas a subnet (#) satisfyi.ng a large dew;ation principle with powers (t). Proof. Define A(h) = logp(e t/t) for ail h E Ch(X). Then. (A,(-)) is a net in the compact space [-et, +cci c(X) (with the product topology), and so there is a subnet (A(-)) converging to some limit A'(-). The result follows from Corollary 4.3 applied to (/t ). [] Remark 4.5. (i) ** (iv) in Theorem 4.1 was known when 3' is a sup-preserving ca- pacity in the O'Brien sense, i.e., 7((7) Ç, (lç) = inf{7(G) : G D Iç.G Ç Ç} for all compact Iç c X, and -), satisfies (2.1) of Lemma 2.1 ([3]). Thus, Thcorem 4.1 removes the SUl)-preserving as well as the capacity conditions of-y (notice the difference between (iv) in Theorem 4.1 and (ii) in Proposition 2.3). In the spirit of Remark 3.5. this means that (It,) satisfies a large deviation principle with powers (te) if and only if (pâo) has a narrow set-theoretic limit in the set {7 G F : lim/(G) = ")(Ui Ci) for all increaing nets (Gi) in Ç}. Remark 4.6. When X is Polish and (p) Bell ([1], Theorem 2.1) implies the equivalen«e of the following statements: (i') (l) satisfies a large deviation prin«iple with powers (l/n) and tight rate fimction: (il') A(h) exists for all h Ç Ch(X), and infr A(hr) = A(h) for each decreasing sequence (bru) in Ch(X) converging to h Ch(X). The equivalence (i) = (il) together with the last assertion in Theorem 4.1 tan be seen as a free tightness analogue of that, by replacing "decreasing" by 'increasing" in (ii'), and removing "tight'" in (i'). Remark 4.7. The relation (4.2) in Theorem 4.1 generalizes a well-known expression of a rate function J for (#tâ), obtained with 1 = J and F ranging over all singletons ([2], Theorem 4.1.18). l EFERENCES 1. W. Bryc and H. Bell. Variational representations of Varadhan functionals, Proc. Amer. Math. Soc., 129 (2001), No. 7, pp. 2119-2125. MR 2002b:60040 2. A. Dembo and O. Zeitouni. Large deviations techniques and applications, Second edition. Springer-Verlag, New York, 1998. MR 99d:60030 3. G. L. O'Brien and W. Verwaat. Capacities, large deviations and loglog laws. Stable Processes and Related Topics (Ithaca, NY. 1990), pp. 43-83. Progr. Probab. 25. Birkhiuser, Boston, MA, 1991. MR 92k:60007 DEPARTMENT OF I,IATHEMATICS, UNIVERSIT5 OF SANTI.GO OF CH1LE. BERNARDO O'HIGG1NS 3363. SANTIAGO, CHILE E-mail address: hcomma_n@usach, cl TttANSACTIONS OF THE AMEttlCAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2925-2948 S 0002-9947(03)03256-2 Article electronically published on February 25, 2003 INTEGRATION BY PARTS FORMULAS INVOLVING GENERALIZED FOURIER-FEYNMAN TRANSFORMS ON FUNCTION SPACE SEUNG JUN CHANG, JAE GIL ('ttOl, AND DAVID SKOUG ABSTRACT. In an upcoming paper. Chang and Skoug used a generalized Brow- nian motion process to define a generalized analytic Feynman integral and a generMized analytic Fourier-Feynman transform. In this paper we establish several integration by parts formulas involving generalized Feynman integrals, generalized Fourier-Feynlnan transforms, and t|le first variation of funclionals of the form F(x) ---- f((¢l,X),...,(¢n,X)) where (¢,x) don(,tes the t'aley- Wiener-Zygmund stochastic integral fo r c( t )dx( t ). 1. INTRODICTION In [11], Park and Skoug, working in the sclting of one-parameter \Viener sl)a.ce C0[0. T] established several integra.ti(m t)y parts formulas involving analytic Feyn- man integrals, Fourier-Feynnmn transfonns, and the first variation of flmctionals of the form (1.1) where (.x) denotes the Paley-Wiener-Zygmund sto«hastic integral In this paper, we also studv flmctionals of the form (1.1) but with :r in a very general function spa«e Ço,b[O,T] rather than in the Wiener space Ç0[0. T]. The Wiener process used in [11] is flee of drifl and is stationary in rime while t.he stochastic process used in this pa.per is nonstationary in rime, is subject to a drifl a(t), and can be used to explain tlw position of the Ornstein-Uhlenbeck process in an external force field [10]. It turns out, as noted in temark 3.1 bclow, that including a drifl terre a (t) ma.kes est.ablishing various integration bv parts fonnulas for Fourier-Feynumn transforms very diflïcult. By choosing a(t) = 0 and b(t) = t oll [0, T], the function space C,b[O, T] rednces to the Wiener space C0[0, T], and so the results in [11] are i,nmediate corollaries of the results in this paper. For related work see [3], [4], and [6]. Received by the editors September 6, 2002 and, in revised form, November 15. 2002. 2000 Mathematies Subject Classification. Primary 60J65, 28C20. Key words and phrases. Generalized Brownian motion proeess, generalized analytic Feynman int.egral, generalized analytic Fourier-Feynman transform, first variation, Cameron-Storviek type theorem. The present research was conducted by the research fund of Dankook University in 2000. (2003 American Mathematical Societ_ 2926 SEUNG JUN CHANG, JAE GIL CHOI. AND DAVID SKOUG 2. DEFINITIONS AND PRELIMINARIES In this section we list the appropriate preliminaries and definitions ri'oto [5] that are needed to establish our parts forinulas in Sections 3, 4 and 5 below. Let D = [0, T] and let (t, B. P) be a probability measure space. A real-valued stochastic process Y on (-, B, P) and D is called a generalized Brownian motion process if l'(0, w) = 0 ahnost everywhere and for 0 = t0 < t < --- < t, _< T, the n-dimensional random vector (Y(tl, w),-.- , Y(t,,w)) is normally distributed with the density function Iç({:,,ï) = ((2r) ' H(b(tj)-b(tj_,))) -/2 (2.1) { 1- ((qj-a(tj))-('tlj-l-a(tj-1)))2} -exp - b(t)-b(t_) where -q = (q,-.-, q,,), q0 -- 0, ï= (t,.-.,t,), a(t) is an absolutely continuous real-valued function on [0,] with a(0) = 0, a'(t) L[0, T], and b(t) is a strictly increasing, continuouslv diffcrentiable real-valued function with b(0) = 0 and b(t) > 0 for each t G [0, T]. As explained in [13, pp. 18-20], Y induces a probability measure p on the measur- able space (]D 13 D where ]I Dis the space of all real-valued functions x(t), t G D, and D iS the smallest a-algebra of subsets of ]D with respect to which all the coordinate evaluation maps et(x) = x(t) defined on ]Dare measurable. The triple (]D BD p) is a probability measure space. This measure space is called the func- tion space induced by the generalized Brownian motion process Y deternfined by a(-) and b(-). Ve note that the generalized Brownian motion process Y determined by a(-) and b(-) is a Gaussian process with mean function a(t) and covariance function r(s, t) = min{b(s),b(t)}. By Thcorem 14.2, [13, p. 18ï], the probability measure p induced by Y, taking a separable version, is supported by C«,b[O, T] (which is equivalent to the Banach space of continuous functions x on [0, T] with x(0) = 0 under the sup norm). Hence (Ca.bI0, T], B(C«,b[0, T]), p) is the function space induced by where B(C«.b[0, T]) is the Borel a-algebra of C«,b[O, T]. A subset/3 of Ca,o[0, T] is said to be scale-invariam nmasurable [9] provided pB is B(C«,b[0, T])-measurable for all p > 0, and a scale-invariant measurable set N is said to be a scale-invariant null set provided p(pN) = 0 for all p > 0. A property that holds except on a scale-invariant null set is said to hold scale-invariant ahnost everywhere (s-a.e.). Let Lâ,b[O, T] be the Hilbert space of functions on [0, T] that are Lebesgue mea- surable and square integrable with respect to the Lebesgue-Stielt.jes measures on [0, T] induced by a(-) and b(-); i.e., L«,[O,T] = v" v2(s)db(s) < and v2(s)dlal(s) < where lai(t) denotes the total variation of the function a on the interval [0, ri. 2 For u, v G L a,b[O, T], let T (2.3) (u, v)«,b = I u(t)v(t)d[b(t) + lai(t)]. J0 INTEG1RATION BY PARTS FO1RMULAS ON FUNCTION SPACE 2927 Then (-, .). is an inner product on L2,b[O. T] and I[ull,b = v/(u, u),b is a norm on L,b[O,T 1. In partioflar, note that Ilu[I«,b = 0 if and only if u(t)= 0 a.e. on [O,T]. N, rthermore, (L,b[O,T], . l],b)is a separable Hilbert space. Let {j}= be a complete orthogonal set of real-valued flmctions of bounded variation on [0, T] such that {0. (¢»¢«).b = . j = ., and for each v Lâ,b[O. T], let (e.4) ,(t) = (,, ¢),.b¢(t) for = 1,2,.-.. Then for each v Lâ,b[O, T], the Paley-Wicner-Zygmund (PWZ) stochastic integral (v, x) is defined by the formula (.5) (,.) im ff = for all x C«.b[O. T] for which the limit exists: one tan show that for each v c.d0.rl, the PWZ integral (v,x} exists for p-a.e, x g denote the function space integral of a B(Ca.b[O. T])-measurablc fimctional F by Jc whenever the integral exists. are now ready to state the definition of the generalized analytic Feymnan integral. Definition 2.1. Let C denote thc complex numbers. Let C+ = {A C ReA > 0} and Ç+ = {A e C" ¢ 0 and Rea 0}. Let F" C,b[O. T] C be such that for each A > 0, the function space integral Jc ..[0.TI exists for ail A > 0. If there exists a function J*(A) analytic in C+ such that J*(A) = J(A) for all A > 0, then J*(A) is defined to be the analytic fimction space integral of F over C.b[0, T] with parmncter A, and for C+ we write (e.7) «' [1 _Z' [(x)] = *(a). Let q ¢ 0 be a real number and let F be a functional such that E [F] exists for M1 A C+. If the following limit exists, we call it the generalized analytic Feymnan integral of F with parmneter q and we write (.S) «'[1 «2"[()l = im --iq where A approaches -iq through lues in C+. Next (see [5], [7], [1], [8], and [6]) we state the definition of the generalized analytic Fourier-Feynnmn transform (GFFT). 2928 SEUNG .IUN CHANG. JAE GIL CHOI, AND DAVID SKOUG Definition 2.2. For A C+ and y G,b[0, T], let (2.9) Ta(F)(9) = En'[F(9 + x)]. For p (1.2], we define the Lp analvtic GFFT, Tq(p:F) of F, by the formula ( e c+), (2.10 / Tqp:F)(9)= lin, T(F)(9) if it exists; i.e., for each p > 0. k" define the L analvtic GFFT. Tq(l: F) of F, bv the where 1/p + 1/p' = 1. fiwmula (A C+) (2.11) Tq(l: F)(9) = lira Ta(F)(9) A--,-- iq if it exists. We note that flw l <_ p <_ '2. Tq(p: F) is only defined as s-a.e. \Ve also note that if Tq(p: F) exists and if F G, then Tq(p: G) exists and Tq(p: G) Tq(p: F). Next we give the definition of the first variation of a fimctional F on C«,b[O. T] tbllowed by a very fimdaln«ntal Cameron-Storvick type theorem [2] which was es- tablished in [5. Theorem 3.5]. Definition 2.3. Let F be a B(C«,b[O, T])-measurable fu,lctional on Ce.bi0, T] and let w C«.b[O, T]. Then (2.2) aF(zl-a,) = F(.r + h=0 (if it exists) is called the first variation of F. Throughout this paper, when working with 3F(z]w), we will alwavs require w to be an element of .4 where (2.13) A = {u, e C«,b[O,T] " w(t) = z(s)db(s) Ol- some z e L]b[O.T]}. Note that for F(z) of the form (1.1), aF(x[w) acts like a directional derivative in the direction of u,. For example, if f(ul,U2) = exp{3u + 4u2} and F(z) = ((, ), (,.)), thl, 6F(z[w) = [a(a, w) + 4(a2. w)] exp{a(a, z) + 4(a2.z)} = (c, )I, ((, ), (, ')) + (, ,,') f((» «), (, «)). The following notation is used throughout the paper: (2.14) alld (2.15) (u2,b ') = u2(t)b'(t)at = for tt L2.b[O, T]. Furt.hermore for ail A C+, v is real part. u2(t)db(t) alwavs chosen to bave positive INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2929 Theorem 2.1. Let z E L2,b[O, T] be given and fort [0, TI, let w(t) = .1 z(s)db(s). For each p > O, let F(px) be p-integrable on Ce,bi0, T] and let F(px) hat,e a first variation 5F(px]pw) for ail x C«,b[O, T] su«h thal for so'me positive fun«tio,t q(p). in t« ollo'wig euation «ist, t« t tird on« aIso «ists, d «qulit olds: (2.16) nf [SF(lw) ] - anf _ =-q [F(.)<z,«>] - (-@(=,a [(x)]. In fact. for each A C+, the above conclusions also hold for analgtic fwwtion space integrals, ç finish this section by stating a very fimdamental integration tbrmula fin the function stmce C«,b[O, T]. Let {1,"" ,«b} be an orthonormal set of funçti«ms from (L,b[O,], and for j {1,--- ,zt} let T (2.18) 4j = (aj,a') =/o aj(t)da(t) and (2.19) Bj (a,b')= a(t)db(t). Note that Bi > 0 for each j ( { 1, 2,--. , z}, while for each j, A may be positive, negative or zero. Let f- be Lebesgue mesurable, and le F(x) = f({al,X},.--, Then dc (2.20) Ç ,, )_} { (uJ-Aj)2}du, dan = Y(.1,---..)p .... _ . ' =l in the sense that if either side exists, both sides exist and equality holds. 3. INTEGRATION B'T PARTS FORMULAS ON FUNCTION SPACE Let n be a positive integer (fixed throughout this paper) and let {01, - , t't.} be an orthonorlllal set of ftlnctiolls fronl (Lâ,b[O, T], I1" lit,b)- Let m be a nommgative integer. Then for 1 < p < oe, let B(p; m) be the space of all functionals of the form (1.1) for s-a.e, x C«,b[O,T] where all of the kth-ordel" partial derivatives fjl,...,j(Ul,''" ,Un) = fj,....,j(ff) of f: '- are continuous and in LP( ") fOl" k ( {0,1,--. ,rn} and each j ( {1..--,n}. Also, let B(cx:m) be the space of all flnctionals of the forln (1.1) for s-a.e, x ( C«.b[0, T] where for k = 0.1,.-- ,m. all of the kth-order partial derivatives fil,... ,j» (ff) of f are in C0(), the spa«e of bounded continuous functions on that Valfish at infinity. Out first lelnma follows directlv from the defilfitions of 5F(x]w) and B(p: m). 2930 SEUNG JUN CHANG, JAE GIL CHOI, AND DAVID SKOUO Lemlna 3.1. Let 1 <_ p < oc be given, let m bc a positive irteger, let F 6/(p: m) be given by equatio (1.1) and let w be an element of A. Then (3.1) F(xlw) = (oj,w)fj((Ol,£,... , j=l for s-a.e, x Ca,[O, T]. Furthermore. as a function of x. 6F(-]w) (p; m - 1). Lemma 3.2. Let p, m ad F be as ir Lemma 3.1. Let z Lâ,[O. T] be 9iven. and for t e [0. T], let w(t) = fO z(s)db(s). Let G e B(p';m) be given by (3.2) G(«) = g((ol,x),-'-, o .-a.«.. z Ca.[0. T]. fi,, (') = F()G() o R (1: m), azd as a flmction ofJ'. 5R(.Iw) (1: m - 1). Proof. Let r(Ul,... ,Un) = f(ul,"" ,Un)9(Ul,''" ,Un). Then (x) --- , (a, x)) is an clement of (1: m) since all of the kth-order partial derivatives of r are contimous and in LJ() for k = 0.1,--- , m. Applying Lennna 3.1 we obtain that R(x[w), as a fimction of x, belongs to B(I: m - 1). Remark 3.1. Let F, G and R be as in Lemma 3.2 above. In evaluating z[n-x)], a,d E[5n-I,)] f > 0. th (3.3) H(A;u,...,u)H(A:ff)=exp{-£(uJ-A)2} =1 occurs, where Ai and Bj are given by equations (2.18) and (2.19) above. Clearly. for > 0, H(A: ff) 1 for all ff G since Bi > 0 for ail j = 1,-.-, n. But for G Ç+, ]H(A;ff)] is hot necessarilv bomded by 1. Note that for each G Ç+, = c +id with c ]d] 0. Hence, for each G H(A;ff) =exp{ - (uj - Aj)29= 2Bj } = exp { [(c2-d2+2«di)u-2(«+di)Ajuj+A] }- , and so (3.5) H(,: ff)] = exp { - - [(c2 - d2) -- 2-cAjuj + A'] } 3 =1 -2Bi " Note that for , C C+, the case we consider throughout Section 3, Re() = c > ]d = hn()[ 0. which implies that c 2 -d 2 > 0. Hence. for each A G C+, H(A; ff), as a function of ff, is an element of LV( ) for all p [1. +]: in fact, H(A: ff) also belongs to C0(). These observations are critical to the development of the integration by parts formulas throughout Section 3. In Sections 4 and 5 below we consider the case where A = -iq +-C+. In this ce,==«+idwith«==]d,. Hen«e, forA=-iq, qeN-{O}. c - d 2 = O, and so (3.6) [H(-iq:ff), = exp{ [Ajuj - A] } =1 2B " INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2931 which is not necessarily ill LP(IR n) for any p E [1, +oc]. Thus, il Sections 4 and 5 we will need to put additional restrictions Oll the functionals F and G in order to obtain the corresponding parts formulas iuvolving Fourier-Fevnmm transforms. Remark 3.2. Note that in the setting of [11], a(t) = 0 and b(t) = t on [0, T] and so Ai = (ctj,a') = 0 and Bi = (ct,b') = 1 for all j e {1,2,-.-,r}. Hence, for all IH(A:/7)I= exp - u =exp u <1. 2 -- j=l Theorem 3.3. Let z e L,b[O.T ] be given and fort e [0. T], let w(t) = f z(s)db(s). Let p, m, F and G be as in Lemma 3.2. Then for all A C+, (3.7) where x/ï is chose to have positive real part. Proof. First define R(x) = F(x)G(x) an, t let r(a,-.-,,) : f(u,.---.an)g(,q,"". ,). Then by Lemma 3.2, R E B(1; m) and (R(.[w) /3(1: m- 1). Furthermore ail of the kth-order partial derivatives of r are contimous and lu LI(iR n) for k = 0, 1.--. , m. Hence, R(px) is p-integrable on C,,b[O, T] for each p > 0. In addition, for s-a.e. x e C.b[O. T], 5R(x[-w) = F(x)SG(xlw ) + 5F(x[w)G(x) = f((Ol,X),''" , (Otn,X)(Otj,W)j((OI,X),''" , (3.8) =1 + g((Ol,X),"" , (On,X)) '(Oj,w)fj((Ol,«),'", (an,x)). j=l But for ail u e L.b[O, T], (3.9) Jô0 T .()d,() u(s)z(s)db(s) T < fo Iu(s)z(s)ld[b(s) + lai(s)] In particular, since {eq,---,cn} are orthonormal, I(» ')1 < Ilzll.. for each 3 e {1,2.--. ,}. 2932 SEUNG JUN CHANG, .IAE GIL CHOI, AND DAVID SKOUG Next, using (3.8) and (3.9), we see that for p > 0 and h > 0. <_ pllzll,»lf(<,,.p., + pi,,,,),..., (o.pa. + ph,v)) i Dut this implios that R(px + phwpw), as a fimction of x. is -integrable for ail p > 0 and h > O. This tan ho seon by intogrting the right-hand side of (3.10) terre by terre. For example, usin (2.20), ve see that for any ! E {1.--- ,}, (3.11) .exi,{_ - [uJ - P(Aj + h(aJ'w})]}du ...du n j= 2p2 B. Thus, using (3.10) and (3.11). we ol)tain that for p > 0 and h > 0. Next, using (3.8), (2.19). (3.3), and (3A), we see that for all A > 0, (3.12) = [/() (l. )l() + () (,, )1,()] (: ). But, as noted in Remark 3.1 above, for each A C+, H(A;/) is an element of C0(), and so the integrand on the right-hand side of (3.12) is in L(). Hence. eF [R(-I,)] = eF [F(')6(I) + F(I)()] exists for all C+. A simil argument shows that the analytic function space integral E[F(x)G(x)] also exists for all C+. Equation (3.7) nowfollows from Theorem 2.1 above; in particular, ri'oto equation (2.17) with F(x) replaced with R(x). INTEGRATION B PARTS FORMULAS ON FUNCTION SPACE 2933 The following two corollm'ies are special cases of Theorem 3.3. Corollary 3.4. Let z, w, and m be as in Theotvm 3.3. Let F B(2: m) be given by (1.1). Then for all A C+, (3.13) AEn x = _ [(F(x)) (z,x)]- (:,a « [(F(x))/. Proof. In Theorem 3.3, «hoose p : 2 and Gx) Corollary 3.5. Let z and z2 bc elem«7,ts of L.b[O.T], and for t [O.T], let wj(t) : fO zj(s)db(s) for j {1.2}. Let F (2:,n) be given by equation (1.1). Then ,for all A C, (3.14) = AEaxn[F(a')SF(xlu,1)(z,x)]- V/--(z,a')En[F(a.)(F(X[Wl)]. Pro@ Let I' = 2 and G(x) = F(:rl., ) in Theorem 3.3. Lemlna 3.6. Let p. m and F be as in Lemma 3.1 obove. Then for all A C, (3.15) for s-a.e. 9 C,b[O. T] "u,herc (3.16) ,. = f(ff + OH(A: )dg[ with t?j and H given by eqaations (2.19) and (3.4) respectively. Pro@ For A > 0, equation (3.15) follmvs easilv fronl equation (2.20). But for each A C+, as shown in Remark 3.1 above, H(A; u,--- ,u,) is an element of L()aC0(l ) for all p [1. oe]. Hence, for each A C+ and s-a.e, y f(//1 -[- (Ctl,/},---,/t n n c (Ctn,//))//():U1, --./An) belongs to L(NI ') and so equation (3.15) hohls throughout tE+. Our next lemma follows from standard results for convolution products. The key is that fol each A C+, H(A: ff) is an element of L»(]K ') ç C0(]K n) for ail 1 <_p<_ +oc,. Lemma 3.7. Let cho (a) If f L (]R ). (b) If f LP(IR ) where p' = P p--l" (c) If f be given by equatioz (3.16) above. then ¢0(A; ") C0(]R ') for all A C+. for some p (1, ac), then b0(A:-) LP'(IR ") for all A C+ then bo(A;-) LI( n) for all A C+. Our next theorenl follows immediately frolll Lemnla 3.7. Theorem 3.8. Let 1 <_ p <_ 2934 SEUNG JUN CHAN(], JAE GIL CHOI, AND DAVID SKOUG Theorem 3.9. Let 1 < p < oe and w A be gwen. Let F B(p; m ) be gzven by «quatzo, (1.1). Then for all A C+ and s-a.e, y C«b[O,T], (3.17) , W) fl(al -- /l, y), - - - ,1tri -- -H(A;al,'-- ,an)dul...dun = T(5f(-I,,))(v). which, as a function of g. is an element of B(p'; m- 1). Proof. The fact that 5T.x(F)(y]w) is an element of /3(p';m- 1) follows directly from Theorem 3.8 and Lemma 3.1. To establish equation (3.17) for A > 0. simply ca.lculate 5T(F)(y]w) using equation (3.15), and then calculate T)(5F(.Iw))(y) using equations (3.1) and (2.9). Finally. equation (3.17) holds throughout C+ by analytic continuation in A. [] In our next theorenl we obtain an integration by parts formula involving Tx(F) and T,(G). Theorem 3.10. Let p. m, z, w, F and G be as in Theorem 3.3. Then for all A C+, (3.18) E" [T(F)(x)ST(C)(xlw) + 5T(F)(x]w)T(C)(x)] = AEï'[T,(F)(x)T(G)(x)(z,x)]- OE(z,a')E''[T,(F)(x)T,(G)(x)]. Proof. For x Ca,b[O.T], let R(x) = T(F)(x)T(G)(x). Then by Theorem 3.8, T(F) /3(p'; m) and T(G) /3(p: m). Hence, bv Lemma 3.2, R belongs to (1; rn), and so by Lemma 3.1.5R(x[w), as a fimction of x, belongs to (l'm- 1). Thus, equation (3.18) follows from Theorem 3.3 with F and G replaced by T,(F) and T.,, (G) respectively. [] Theorem 3.11. Let m,z and w be as in Lernma 3.2. Let p [1.2] and let F and G in B(p:m) be given by equations (1.1) and (3.2) r«sp«ctively. Then for ail AC+, E.'[F(x)A(C)(xlw) + F(xlw)T(C)(x)] (3.19) = t'[F(.)T()(.)(,.)]- OE( -"-',, z,. )r.. [F(x)T.(G)(x)]. Proof. Let R(x) = F(x)T(G)(x) for x Ca.b[O,T]. Bv Theorem 3.8, T(G) is an element of/3(/;m) and hence bv Lemma 3.2, R belongs to /3(1; m). Hence, by Lemma 3.1, 5R(x]w), as a function of x, belongs to/3(1"m - 1). Thus, equation (3.19) follows from Theorem 3.3 with (7 replaced bv T,((7). [] Corollary 3.12. Let m, z, w,p and F be as in Theorem 3.11. Then for ail A C+, (3.20) E'[F(x)5T(F)(xlw) + 5F(x[w)T.(F)(x)] = E'[:()T(F)(.)(z,.)]- OE(z..')F[F(.)T(F)()]. Pro@ Simply choose G = F in Theorem 3.11. [] INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2935 Corollary 3.13. Let m, z and w be as zn Lemma 3.2. Let F E /3(2;m) be given 2 by equation (1.1). Then for all A C+, Eoe n: [TA(F)(x)6T(F)(x]w)] (3.21) A __-- __ Ean. 2 [(TA(F)(x))2<z'x] Proof. Siinply choose p = 2 and G = F in Theoreln 3.10. 4. PARTS FORMULAS INVOLVING Tq(l: F) AND Tq(l: G) In this section we obtain various integration by parts fornmlas involving the analvtic GFFTs Tq(1; F) and Tq(l: G). In view of equation (3.6) above, we clearly need to impose additional restrictions on the functionals F and G than were needed throughout Section 3. Fix q e - {0}. Then as A -iq through values in C+, « = Re() /2 and Idl /2 where d = hn(). Next using equations (3.3) through (3.6) we see that for all A G Ç+ with c = Re() < ((1 + ]H(A;ff) =exp{- (4.1) G exp <_ exp In addition. (4.2) 1 For f La(IR ') let (4.3) 9r(/)(O = (27r)- denote the Fourier transform of f. Theorem 4.1. (1.1) with (4.4) , f(ff)exp { i Z ujçj }d j=l Let q IR - {0} be given. Let F /3(1; m) be given by equation 1 2936 SEUNG JUN CHANG, JAE GIL CHOI, AND DAV|D SIxOUG for all k ¢ {0, 1,--. ,m} and each j {1,--- .o}. Furthermore, assume that belongs to Co(lI). Then (4.) ¢o(-«:4) --_ _ -- f((+ ff)H(-iq: )d6 is an, element of Co(II). Furthermore, the L1 analgtic GFFT, Tq(1; F) exsts as an elemeot of/3(c: m) and for s-a.e, g Ce,bi0. T] is givet by tbe formula (4.7) Tq(I: F)(y) = ¢0(-1q; <al, g),'- , <cn, g)). Proof. By (4.1) and (4.4) we know that f(.)H(-iq: .) L(]n). and so its Fourier transform, ,T(f(.)H(-iq:.))(O cxists and bclongs to C0(]). Furthermore, bv cquations (4.6) and (3.4) and thc fact that vfZiq = c+ di = V/2 + di, we obtain 1 (4 8) - exp J= 2Bi B " ' Bn " Bv assumption (4.5), it follows that d)0(-iq: 0 is an element of Co(n). Finally, by equations (2.11), (3.15), (3.16), (4.8) and the dominated convergence theorem (the use of which is justified by (4.2)), it follows that for s-a.e, g INTEG1RATION B PARTS FORMULAS ON FUNCTION SPACE 2937 Theorem 4.2. more. assume that for each 1 { 1, 2,-.- , n }, Let q IR - {0} and F B(I m) be as m Theorem 4.1. Further- { } ( ) q'l q n (4.10) exp - 9-[qAjJ (fl(.)H(--iq;.))- , ..... belongs to Co(IRn). Tken for eack 1 {1,2,---,n}. (4.11) çt(-iq:() = _ , ff((+ ff)H(-iq:ff)di[ is an element of Co(Rn). In addition, for each w A and s-a.e. 5Tq(I: F)(y]w) = (t, w}Ot(-iq: (a,y), . - - , (a.,y)) (4.12) =l = (. aF(.I))(v), which, as a fuc*io of g. is a element of B(;m- 1). Pro@ The proof that each çbt(-iq; .) belongs to Co(IR ') is the saine as the proof in Theorem 4.1 above showing t hat ¢0 (-iq:') Co (IR '). Equa.t ion (4.12) t hen follows immediately using the defilfition of the first variation and equa, tion (4.7). [] Our next theorem gives a parts formula involving F and Tq(I: G). Theorem 4.3. Let q - {0} be given and let F B(I: m) be as in Theorem 4.1. Let G B(I: m) be given by equation (3.2) with (4.13) for all k {0, 1,--- ,m} (4.14) exp { - -Ç 1 2 =1 and each ji {1,--. ,ï}. Furthermore, assume that 2X/-[Ajj } -(gl(.)H(_iq: .)) ( _ ql 2938 SEUNG JUN CHANG, JAE GIL CHOI AND DAVID SKOUG belongs to Co(N n) for all l t e [O,T], let w(t)= f z(s)db(s). Then Lêt z L,[O.T] be gwen and for E'fq[F(x)STq(1; G)(xlw ) + 5F(x[w)Tq(1; G)(x)] (4.15) = -iqEnfq[F(x)Tq(l: G)(x)(z,x)] - (-iq)(z.a')E'f[F(x)Tq(l'G)(x)]. Pro@ Let R(x) = F(x)Tq(l'G)(x). By Theorem 4.1. Tq(l:G)(x) is an element of B(; m) and so R(x) is an element of B(I: m). Also, by Theorem 4.1, Theorem 4.2 and Lemma 3.2, 6(x,,,) = F()6T(I: (1,) + 6F(,)T(: a)(x). as a fimction of x, is an element of B(I" m - 1). In addition, we know that for each I e {0.1,.-. ,.}, 1/2 Je[- 9(ff+ ff)H(-iq:ff)dff /'t(-iq:ç = _ 27rBj is an element of C0(IR n) with and for s-a.e. exist: (4.16) and Tq(l:G)(y) = bo(-iq: (c,y),-.-, 6Tq( l: G)(y]w) = Z (ct, w)g't(-iq; (c,, y), . . . , (c,, y) ) /=1 y E C,,b[O. T]. Hence, both of the following analytic Feynman integrals Exanfq[/(X)] = Enfq[F(x)Tq(l'a)(x)] = ( I @ ) /2 ,, f(ff)bo(-iq: ff)H(-iq', ff)dff j=l Exanfq [(!(X1732)] = Enf[F(x)STq(1; G)(xlw ) + F(xlw)Tq(l: G)(x)] - -(,-iq)/£[ (4.17) -- + 0(-iq:ff) (at,w)ft(ff) H(-iq:ff)dff. /=I Also, proceeding as in the proof of Theorem 3.3 above, it is easy to show that for p>0andh>0, E[[6(px + pbwpw)] (4.18) _ + /:1 INTEGRATION B PARTS FORMULAS ON FUNCTION SPACE 2939 Hence, by Theorem 2.1 above, the analytic Fe.wmmn integral E'G[R(x)(z,x)] = exists and equality (4.14) holds. [] Choosing G = F in Theorem 4.3 we get Ihe following integration by parts for- lnula. Corollary 4.4. Let q C OE- {O} be given and let F B(1;m) be as i Theorem 4. `) . Let z ad w be as in Theorem 4.3. Thez Eaznfq [F(:r)STq( l : F)(x[w) + F(x[w)Tq(1; F)(.r)] (4.19) = -iqE"f[F(.r)Tq(l: F)(x)(z,x)] (-iq)½(z, ,,a,f - a lr, [F(.r)Tq(l:F)(z)]. Next we obtain a parts formula involving Tq(l F) and Tq(l: G). Theorem 4.5. Let q OE- {0}. Let F B(I: m) be as in Theom 4.2 and let G B(I: m) be as in Theorero 4.3. Furhermore, assume Hmt .for each l {0.1,--- n}, (4.20) 1, .@(-iq: ff)H(-iq; ff) dg Then for w(t)= fô z(s)db(s) with z (4.2) Pro@ Let R(x) = Tq(l:F)(z)Tq(l:G)(x). Then R G B(oc,: m) and 6R(.rlw ), as a function of :r, is an element of B(oc;m- 1). Hence, by (4.6), (4.11) and (4.20), both of the following analytic Feymnan integrals exist: (4.22) and (4.23) 1 j=l b0(-iq: ff)ç'0(-iq; ff)H(-iq: ff)dg _ 2rrBj i 13. qS°(-iq: g) E(at" w)@(-iq', g) n 1=1 ] + bo(-iq; ff) E(oq, w)@(-iq;) H(-iq:)dff. /=1 In addition, for p > 0 and h > 0, < pllzllo.» [],;bo(-iq; )ll, II@(-iq; )11, (4.24) /=1 + II'b0(-/q; ")lloe E Ilçl(-iq; 2940 SEUNG JUN CHANG, JAE GIL CHOI, AND DAVID SKOUG Hence, bv Theorem 2.1, the analytic Feymnan integral E [R(x)(z, )] exists and equality (4.21) holds. finish this section with some examples which shed light upon the necessity of conditions such (4.4) and (4.5), and which also illustrate that the conclusions of Lemma 3.7 are hot necessarily valid for A Ç Ç+ with e() = 0. In our first exainple ve define a flmctional F of the form (1.1) with n = 1, such that F is an elelnent of B(p; m) for ail p Ç [1, +], f is an element of L»(N) for all p [1, +], and yet o(i; ") given by (4.6) is hot an element of Co(). In fact, [o(i:)1 = +OE for all ( Ç N. Exalnple 4.6. Let q = -1, let 't = 1, let mbe a nonnegative integer, and let be an elelnent of La»[0, T] with IIllla» = 1. Without loss of generality, we will smne that ,4 (see equation (2.18)) is positive. Let f " N + C be defined bv the fornmla {iuî iA,u, A AlUl } (4.25) I(,) ç+Xo,+)()p 2e 2 + 2 4 " note that 2B1 4B " and hence f Ç LP(N) for all p Ç [1, +oe]. In fact, f is also an element of Co(N). ç then define F C,[0. T] C by the formula (.) F() I((,, )). It is easv to see that F Ç B(p: m) for all p Ç [1, +]. Next, using equation (3.4) with = 1, A = i, and _ +i, we observe that (4.28) H(i; ul) = exp { Aul + iAu - A - iuî } 2B " and hence (4.29) f(Ul)H(i:l)=U+'[o,+)(l)exp{ A } 4B1 " which is not an element of Lp(N) for anv p Then, using equation (4.6) with n = 1 and q = -1, equation (4.25) and equation (4.28), we s that 1 () (i;) = 2B f(u +)H(i;ul)du 1 = 2ÇB exp 2B (4.30) - (u +)+k[o,+)(u +)exp iu + du B 4B 1 () {«î « = exp 2B 2B - (. )+ ex + . ç B 4B1 INTEG1RATION BY PARTS FORMULAS ON FUNCTION SPACE 2941 Thlls, (4.31) /_t-(x {tLtll v/Alttl } ] (Lt 1 ÷ 1) m+l exp -- + du.1 . 1 B1 -lB1 Hcnce, choosing 1 - 0, and using the fact that Al is positive, we see that lao(i: O)l = (2B,)- u m+x exp : (2gB1) - 0 m+l exp du = +, which implies that 0(i; ") is not an element of C0(R). hl fa«t. for each fixed R, w observe that ,0(/;,),=(2B)_exp { .4,1}4B1 (4.32) (Ul+ exp + dul = and so 00(i; ") is hot an elelnent of LP(N) for any p [1, +] even though f(.) was an elelnont of LP(N) for all p [1. +] and F was an elclnent of B(p: m) for all p [1, +oe]. Hence, the L1 analytic GFFT, T_(I: F) does hot exist. ç also note that f does hot satis6" condition (4.4) above since by equation (4.26) (recall that q =-1 and so ()/2 = 1), lf(ul)]exp { IAlUll } dulB "tlUl } v"2.4ul + __ dal = +oc. 4B1 BI In our lmXt example we exhibit a fUlctional F of the forln (1.1) that satisfies conditions (4.4) and (4.5) above. Furtherlnore, we are able to evaluate the integral in equation (4.6) and thus obtain a forlnula for ¢0(i: ) which does not involve anv integrals. Example 4.7. Let q = -1, let m be a nonImgative integer and let r be a posi- tive integer. Let {ai,---, a} be an orthonolnml set of functions froln (L.b[0. T], 1" [«,b). and for ea.ch j e {1.---.n} let Ai and B be given by (2.18) a.nd (2.19) respectively. define f : N Cbv the forlnula [u - iAuj + A - u -- (4.33) f() exp . J= 2B }Ve note that (4.34) j=l 2Bj and hellCe f e LP(] n) for ail p e [1, +o1. Also, f e Co(IR). 2942 SEUNG JUN CHANG, JAE GIL CHOI, AND DAVID SKOUG Let F " C,.b[O, T] --, C be given by (4.35) F(X) f((l,x),"" ,(a.,x)). It is easy to show that F B(p; m) for all p [1. Next, using equation (4.33), together with equation (3.4) with A = i and x/ï = 1+{ it follows that (4.36) f(ff)H(i; ff) = exp { - uj Now clearly f(.)H(i: .) is an element of LP(N ") 0 Co(N ") for ail p [1. +oe]. Next, using equations (4.6), (3.4) and (4.33) we obtain (4.37) f exp i Ujj 1 j:l because (4.38) HClIC% (4.39) Ioo(i;()1 =exp- 2Bi ' j=l and so 00(i; ") is an element of C0(lI n) N LP(I n) for all p e [1, +oc]. 2 We also note that because of the factor exp{ 2--ï, } in the definition of f(ff) given by equation (4.33), condition (4.4) certainly holds. In addition, condition INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2943 = Bj 1¢0(i; Hence, by Theorem 4.1. the Lt g 5. PARTS FORMULAS INVOLVING Tq(2;/) AND T(2:G) Note that in OllI" first theorem below we replace conditions (4.4) and (4.5) with condition (5.1). This condition is used to obtain a dominating flmction in order to apply the dominated convergence theorem. Theorem 5.1. Let q IR - {0} be given. (1.1) with Let F B(2: m) be given by equation for all k {O. 1,... ,,n} and each j {1.---,,z}. Then 1 (5.2) ¢o(-iq;& = _ 2rrB) ,, f((+ ff)H(-iq: g)dff is an element of L2(N). Furthermore. the L2 analytic GFFT. Tq(2: F) ests as an element of B(2ç m) and for s-a.e, y C,b[O, T] is given by the formula (5.3) Tq2: F)(y) = ¢o(-iq; (c, y),. -- , (a,, y>). Proof. Using (4.1) we first note tllat If((+ ff)H(-iq: ff)ldff f((+ff)l exp {(1+}q1)2 J= Bj 1 léo(-iq()l-< _ 2-7-D-7) 1 j= 2roB9 ] dff. 2944 SEUNCI JUN CHAN(3, JAN GIL CHOI, AND DAVID StxOUG Hence. by (5.1) with k = 0. iàlld so 00(--iq;() is ill elen,t'nt of L2(lIr). To show that Tq(2: F) exists and is given by equation (5.3) it stlflîces to show that for ea«h p > 0. }ira /[ TX(pq) -cO(-iq; (o,.p./),'" , ((rt. py))12dl(l) -\--q. »[0,T] But No,v clearlv 4,0(A:) 00(-iq:) a.e. o,, IR as A -- -iq through vah, es in C l . Tlms, to show that 114,o(,x; .) - 4,o(-iq;-)112 ---' o, it suffices [11, p. 126] to show that 114,0(,x: )112 --+ 114,0(-iq: )112 asA -.iq through vah,es in C+. But forall A G C+ with Re(v/ï) < ((l+lql)/_)-. "9 j=l INTEGRATION BY P,RTS FORMULAS ON FUNCTION SPACE 2945 Hence, b¥ the dominated convergence theorem, lira 0H(A: 2 5")dff 2 f(ff + OIq(-iq; Corollary 5.2. Let q E IR - {0} and F /3(2; m) be as in Theorem 5.1. Then for each ! { 1,2,..- , n}. 1 j=l is an element of Lu(N'). In addition, for ea«h w e A and s-a.e, g e Ce,bi0. T], (5.5) 5Tq(2; F)(ylw ) = Z(a' w)¢,(-iq; (a,,g),-.- , = Tq(2:5F(.Iw))(y ), which, as a function of g, is an eleme'nt of/(2: m- 1). ,)) Pro@ The proof that each ¢(-iq;-) belongs to L2(IR n) is the saine as the proof in Theorem 5.1 above showing that ¢0(-iq-') Lu(IR'). Equation (5.5) then follows inmmdiately using the definition of the first variation and equations (5.2) and (5.3). [] Theorem 5.3. 5.1. Furthermore, assume that (5.6) [f ),,...,j (ff)H(-iq: ff)[dff < oo for all k e {0,1,---,nz} and «ach ji e {1,2,--. ,n}. Le G 8(2; m) be 9iven b equation (3.2) with 1 (5.7, £, [£ }gj,.....j«((+ff, lexp{ (1 ,q,) »r all k e {0,1,--. ,m} and «ach ji e {1,---,n}. 4.3. Then (5.s) Let q IR - {0} be given and let F /3(2:m) be as in Theorem IAufil d d( < oc Let z and w be as in Theorem 2946 SEUNG JUN CHANG, JAE GIL CHOI, AND DAVID SKOUG Proof. By Theorem 5.1, for each I E {0, 1,--. ,n}, 1 (5.9) g,,(-iq;()= (fi) j=l t, gt(( + ff)H(-iq; ff)dff alld Exanfq [/(X)] = 27rBj Also, for p > 0 and h > 0, <_ pl[z[I«»( 1-I 2P2B) -½ IIf[12 ]]t(-iq;-)]12 3=1 /=1 + ][o(-iq; ")112 liftIl2 l-1 /(ff)H (-iq: ff)Z/'0 (-iq: ff)da £ f(ff) + "0o(-iq; g)Z(at, w)ft(ff) H(-iq:)d. l=1 Hence, by Theorem 2.1, the analvtic Fevmnan integral E'q[R(x)(z,x)] exists and equality (5.8) holds. [] Next, choosing G = F in Theorem 5.3, we obtain the following integration by parts formula. Corollary 5.4. Let q. F B(2: m), z, and w be as in Theorem 5.3. Then Enf[F(z)6Tq(2; F)(zlw) + 6F(zlw)Tq(2; F)(z)] = -qEnf[F(z)Tq(2; F)(x)(z,z)] - (-iq) (», ')z2e [F()T(: is ai1 elelnent of L 2 (11 ). Furt hernlore, (5.10) Tq(2: G)(x) = b0(-iq; (al.x),'." , (,x) ) is an clement of B(2; m), raid as a flmction of x, (5.11) /-----1 belOllgS to (2;m- 1). Hel,Ce, /(«)= F(«)Tq(2:G)(x)is al, elenmnt of B(l:m) alld tiR(xlw) = F(z)6Tq(2: G)(.rlw) + 6F(xlw)Tq(2: G)(x) is ai1 eh?lllPnt of B(1; m,- 1). Since f(ff)H(-iq; ff)bt(-iq:ff) and ft(ff)H(-iq: ff)b0(-iq: belong to L(1R ) for eaçh / Ç {0, 1,-.- , u}, both of the following analvtic Fevmnan illtegrals exist: INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2947 Our final theorem is a counterpart to Theorem 4.5 above. Theorem 5.5. Let q C ]-{0} and let F and G be as in Theorem 5.3. Furthermore, assume that for each l {0, 1,--. ,n}, ./ ]¢,(g)(-q: )]d < . (5.12) Let z G L,b[O,T] be gwen and fort e [O,T] let w(t)= fô z(s)db(s). Then Efq[Tq(2: F)(.r)STq(2: G) (x[w) + 5Tq(2; F)(.rlW)Tq(2: G) (x)] -qE [Tq(2; F)(x)%(2:G)(x)(z,x)] (5.13) = - anf (--iq) (Z, t,--anfq - « )x [Tq(2: F)(x)Tq(2:G)(x)]. ProoI. Let R(x) = Tq(2:F)(x)Tq(2:G)(x). Then R G B(1;m) and 6R(-[w) G E " (x B(1;m- 1). Using (5.2), (5.4), (5.9) and (5.12), we sec that [ )] and E"[6R(x]w)] both exist and are given bv equations (4.22) and (4,23) rcspec- tivclv. Finally, we sec that (5.13) follows from Thcorem 2.1, since for p > 0 and h>O, We finish t his paper wit h some very brief couunents about the functionals defiued in Examples 4.6 and 4.7 for the case p = 2. We first note that for the functional F(x) B(2; m) defined by equation (4.27) with f(u) L2(N) given by (4.25), the L2 analvtic GFFT, T_(2;F) does not exist because I¢0(i;)1 = +oe for each N. In fact, the Lp analytic GFFT, T_(p; F) does not exist for anv p [1, 2]. On the other hand, it is quite easy to see that condition (5.1) holds for the fun«tion f(ff) given by equation (4.33). Hen«e, for F(x) defined by equation (4.35), the L2 analytic GFFT, T_(2:F) exists as an element of B(2;m) and for s-a.e. y C«,b[O,T] is given by the right-hand side of equation (4.41). In fact, for all p C [1,2], the Lp analytic GFFT, T_(p: F) exists as an element of B(p';m) and is given by the right-hand side of equation (4.41). EFERENCES [1] R.H. Cameron and D. A. Storvick, An L2 analytic Fourier-Feynman transforrn, Michigan Math. J. 23 (1976), 1-30. MR 53:8371 [2] __, Feynznan integral of variations of functions, in Gaussian random fields, Ser. Prob. Statist. 1 (1991), 144-157. MR 93b:28035 [3] K. S. Chang, B. S. Kim, and I. Yoo, Fourier-Feynznan transform, convolution and first variation of functionals on abstract Wiener space, Integral transforms and Special Functions 10 (2000), 179-200. MR 2001m:28023 [4] S. J. Chang and D. L. Skoug, The effect of drift on the Fourier-Feynznan transform, the convolution product and the first variation, Panamerican Math. J. 10 (2000), 25-38. 2948 SEUNG JUN CHANG, JAE GIL CHOI. AND DAVID StçOUG [5] , Generalized Fourier-Feynman transforms and a first variation on ]unction space, to appear in Integral Transforms and Special Functions. [6] T. Huffman. C Park and D. Skoug, Analytic Fourier-Feynman transforms and convolution, Trans. Amer. Math. Soc. 347 (1995), 661-673. Il=( 95d:28017 [7] , Generalized transforms and convolutions, Internat. J. Math. and Math. Sci. 20 (1997), 19-32. M1R 97k:46047 [8] G. W. Johnson and D. L. Skoug, An Lp Analytic Fourier-Feynman transform, blichigan Math. J. 26 (1979), 103-127. MR 81a:46050 [9] __, Scale-invariant measurability in Wiener space, Pacific J. Math 83 (1979), 157-176. MR 81b:28016 Il0] E. Nelson, Dynamical theories of Brownian motion (2nd edition), Math. Notes, Princeton University Press, Princeton (1967). MR 35:5001 [11] C. Park, and D. Skoug, Integratzon by parts formulas involving analytic Feynman integrals, Panamerican Math. J. 8 (1998), 1-11. MR 99i:46031 [12] H. L. Royden, Real Analyszs (Third edition), hlacmillan (1988). hIR 90g:00004 [13] .1. Yeh, Stochastic Processes and the Wiener Integral, Marcel Dekker, Inc., New York (1973). MR 57:14166 DEPARTMENT OF IkIATHEMATICS, DANKOOK UNIVERSITY. CHEONAN 330-714. KOnEA E-mail address: sejchangdankook, ac. kr DEPARTMENT OF IATHEMATICS, DANIxOOK UNIVERSITY, CHEONAN 330-714. KOREA E-mail address: jgchoidankook, ac. kr DEPARTMENT OF IIATHEMATICS AND STATISTICS. UNIVERSIT5 OF NEBRASKA. NEBRASKA. 68588-0323 E-mail address: dskougmath.unl, edu LINCOLN. TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY çolume 355, Number 7, Pages 2949-29ïl S 0002-9947(03)03269-0 Article electronically published on lIarch 19. 2003 THERMODYNAMIC FORMALISM FOR COUNTABLE TO ONE MARKOV SYSTEMS MI('HIKO Yi:EI ABSTRACT. For countable to one transitive Markov systems we establish ther- modynamic formalism for non-H61der potentials in nonhyperbolic situations. We present a new method for the construction of conformal measures that satisfy the weak Gibbs property for potentials of weak bounded variation and show the existence of equilibrium states equivalent to the weak Gibbs measures. \'e see that certain periodic orbits cause a phase transition, non-Gibbsianness and force the decoEv of correlations to be slow. We apply our results to higher- dimensional maps with indiffcrenl periodic points. §0. INTRODUCTION Thermodvnainic for|nalism for hyperbolic systems was satisfactorily established with Bowen's program ([2]). The existence of geuerating finite l\Iarkov partitions and analysis of Ruelle-Perron-Frobenius operators associated to H61der potentials allow one to show the existence of torique equilibrimn states that satisfv the Gibbs property (in the sense of Bowen) and exponential deca,v of correlations. Also, the pressure fllnctions are analytic and there is no possibility of phase transition (non- uniqueness of equilibrium stat.es). Furthermore, the analyticity pïoblem is strongly relateà to nnlltifractal problems and the zero of Bowen's equation determines the Hausdorff dimension of linfit sers arising from certain iterated flmctional systems ([4], [8], [12], [22]). On the other hand, phase transition, failure of the Gibbs prop- erty and slow decav of correlations can be observed for many COlnplex svstems which exhibit common phenomena in transition to turbulence (the so-called Iz- terrn{tte«y). In this paper we shall construct mathelnatical models which exhibit such phenomena and for this purpose we shall establish thermodvnanfic formal- ism for non-H61der potentials in nonhyperbolic situatious in the following sense: generating l\Iarkov partitions are countable partitions and dvnamical iustabilitv is sbecponential (subexponential decav of cylinder sizes). More specifically, for count- able to one transitive Markov systelns we shall construct conformal measures oe that are weak Gibbs Ineasures for potentials q of weak bonded variation (WBV)(see clef- initions in §1) and show the existence of equilibriuln states t/ for q equivalent to the weak Gibbs measures oe. The conformal lneasures oe associated to q play im- portant roles as reference measures from the physical point of view, and absolute continuitv of equilibrimn srates allovs one to describe statistical properties of ob- servable pheltoInena in the physical sense. In order to clarifv typical reasons for Received by the editors Match 28, 2002 and, in revised form, Septlember 10, 2002. 9 2000 Mathematics Subject Classification, Primary 8D99, 28D20, 58C40, 58E30, 37A40, 37A30.37C30. 37D35, 37F10, 37A45. (2003 American I\Iathematical Society 2950 M. YURI plmse transition, non-Gibbsianness and slow decay of correlations, we introduce in §4 a notion of idifferet periodic point associated to potentials ¢ of VBV. Those periodic points cause failure of sunnnable variation for potentials ç and failure of bounded distortion of local 3acobians with respect to the weak Gibbs measure u for ¢ (Proposition 4). Then a construction of weak Gibbs measures u for ¢ admitting indifferent periodic points implies subexponentlal mstablht in terres of cylin- der mesures (Proposition 3). Furthermore, a construction of equilibriunl states for such equivalent to oe (Theorem 6, Lenmla 15) allows us to show both phase transition (Çorollary 2 Theorem 8) and non-Gibbsianless of equilibrimn states (Theorem 5). bi particular, our results are applicable to the following piecewise CLsmooth countable to one Markov lllap8 T defined on bomded regions X C u with indifferent periodic points (T%'0 = x0, det DT(.r0)l = 1) for which the p tentials - log det DT] satisfv neither sunnnable variation nor bounded distortion so that previous results cannot be applicable. Exau,ple A. (Inhomoge'neous Diopha,tine approximations [13], [15], [16], [17], [19]. [20], [21]). Let X = {(x,.q) N" 0 g 1,-g .r < -g+l} and define T'XXby 1 - , . -- + ___ ZN). This map admiIs indifferent periodic points (1.0) and (-1.1) with period 2, i.e., IdetDY(1,0)l = IdetDY(-1.1)l = 1 and is related to a Diophantine approxilnation problem of inhomogeneous linear class. complex continued fraction transforlnation T ¥ X on the dimnond-shaped region X = {z = .ra +" - 1/2 G a.a- G 1/2}, where a = 1 +{, by T(z) = 1/z -[1/Zll. Here [z] denotes la, + 1/2]a + [2 + 1/21, where z is written in the form z = aa + ,[] = max{n Z I G x}(« N) and [] = m{n Z I n < z}(a Z- N). This transformation bas an indifferent periodic orbit {1,-1} of period 2 and two indifferent fixed points at { and -i. ç recall previous works related to thennodynanfic fonnaIism for countable to one Markov systems. For countable Markov shifl.s, O. Sarig proved the existence of conforlnal measures and equilibrium states associated to locally H61der potentials defined in [10] and D. Fiebig, U. Fiebig and the author proved the existence of equilibriuln states for potentials satisfying bounded distortion (supnk Çn < OE in the definition of WBV) in [7]. Our main Theorems 4-8 do ilOt satis" these as- sunlptiollS and Examples A, B show that they cannot be treated by methods in [0] and in [7]. rthernlore, for higher-dimelSiolml sy8teillS that are not symbolic systems we mav bave crucial difficulties in verifying the positive recurrence con- dition impod on potentials in both [10] and [7]. The infinite iterated functional systems that Manldin and Urbanski studied in [8] correspond to the local inverses of piecewise confornml countable Bernoulli systems in out sense, and the method nsed in [8] severely relies on the Bernoulli property which fails to hold for Example B. Moreover, a H61der-type condition, imposed on potentials for the existence of con%rmal measures and %r establishing a variational principle, is ilOt satisfied bv the important potentials -log ] det DTI for both Examples A and B. COUNTABLE TO ONE MARKOV SYSTEMS 2951 In order to prove our theorems, we first give iii §2 ail appropriate definition of topological pressure for countalle to one transitive Markov systems with finite range structure. Our definition coincides with the standm'd one by using peri- odic points under certaiu conditions (Lemma 6) which tan be easily verified for higher-dimensional exanlples in §8. We also associate the topological pressure to the spectral radius of the Perron-Frobenius-lRuelle operator (Theorem 3). The es- sential issue for constructing both weak Gibbs measures u for 05 and equilibrium states p. for 05 eqnivalent to u is to derive Schweiger's jump transformations T* over full cylinders (see the definition in §1) with respect to which a local exponential instability (05) and a local bou'nded distortio (06) for potentials ¢ are satisfied. Then showiug the existence of a zero of a generalized Bowen's equation (GBE) for derived potentials ç* associated to T* (Lemnm 7) allovs one to show the existence of conformal measures that are weak Gibbs measures for (Theorem 4). Under a mild condition which cannot be covered bv previous works, we show the existence of a zero of (GBE) in §3 by using a product formula of zeta funct.ions (Proposition 1), which shows a nice relation between zeta functions for the original systems and zeta functions for the jump transformalions. We also construct a-finite conformal measures via induced maps TA over a single full cvlinder .4 (Theorem 7) in §6 by using some idea that appeared in a previous work bv M. Denker and the author [5] in which no evidence of the existence of weak Gibbs confornml measures was given. We establish the existence of equilibrium states p for 05 of WBV equivalent to the weak Gibbs measure u for 05 via a junlp transformation (Theorem 6) in §5 and via induced maps (Lemma 15) in §6. Theu we can immediately see that the appearance of indifferent periodic orbits associated to 05 implies a phase transi- tion, i.e., failure of the uniqueness of equilibrimn states (Corollary 2. Theorem 8). We should remark that our construction via induced maps shows the existence of (countably man3- ) lnutually singular equilibrum states. In §8 we apply our results to higher-dimensiolml piecewise C Markov maps with indifferent periodic points. AI1 proofs of results in §§2-3 are postponed to the Appendix. §1. PRELIMINARIES Let. (X, d) be a compact metric space and let T X -+ X be a noninvertible map that is hot necessarily continuous. Suppose that there exists a countable disjoint partition Q = {-¥i}iÇI of ,¥ such that iÇI intX is dense in X and the following properties are satisfied. (01) For each i I with intXi ¢ O,T[itx, "inlXi T(intXi) is a holneomor- phisln and (T]x,)- extends to a homeolnorphisln ri on cl(T(inlX)). (02) (03) {Xi}ieI generates , the a-algebra, of Borel subsets of X. We say that the triple (T, X. Q = {X}eI) is a piecewise C°-invertible system. By (01), F[tx, extends to a holneomorphisln (v) - on cl(intXi) for i e I with intXi ¢ O. For notational convenience we denote (v) - = Tcl(zntx). Let i = (i ... i) I' satisfy int(X T-1Xi ... T-(n-)Xi) ¢ O. Then we define X := Xii T-1Xi= ... T-(n-1)Xi, which is called a cylinder of tank n and write Iii = - UN (01), TI,,x,1 ..... imX,1...i + ç(int(X...,)) is a homeo- morphism and (Tn]intXil...i)-I extends to a homeonorphism Vil o vi= o ... o vi = Vil...i,"cl(T(int-k)) cl(intXt) and (Vil...in) -1 = rl,,,x...,.» x> impose 2952 M. YU1RI on (T, X, Q) the next condition, which gives a nice comltable states swnbolic dy- nallli£s Sil,lilar to SOfi£ shifts (cf. [15], [161, [171, [19]): (Finite Range Structuv). = {it(T"Xh....) " VXh....,Vn > 0} consists of finitely many open sul)sets U1, .... UN of X. In particular, if (T, X. Q) satisfies the Markov property (i.e., inXi intTAÇ. 0 ilnplies inTXj D idX), then lg = {it(TX) Vi I} and we say that (T, X. Q) is an FRS Markov sys*«w.. If Ni Q satisfies cl(T(i*Xi)) = X, then Xi is called a ,[vll clinder. If all cylinders arê full cvlinders so that = {iLY}, then (T, X. Q) is called a Beruoulli sWtem. " assulne further the next transitive condition: N (Transitivitg). btX = U= ut and Vl E { 1,2 .... , N}, 30 < s < OE such that for each k E {1,2,..., OE'}, U contains an inlerior of a cylinder ,Y(t0(s) of rank s such that T ' (itX(k')(sz)) = U. The transitivity condition allows one to establish the next fact. Lenllna 1. Therc exists (} < 5' < su«h that Ts(Uï= intX(k'O(s)) = intX and VXi Q. TS+Z(idX,.) = intX. Pro4 4 L«mma 1. Since each Uk contains Uï=] intX(k'l)(,), choosing S = st is enough t.o establish the desired fact. Re'mark (A). If (T, X. Q) is a Markov systeln, lhen lhe transitivity condition implies «periodicitg in the following sense: S > 0 such that VU, U H,V > S. = (iZ ...in) with itX ¢ satis{ving btXi, C Uk and T(int(Xi,)) = U. Definition. } say that is a potential of weak bouuded variation WBV) if there exists a sequence of posit ive lmmbers { Cn } sat isfying lim (1/ ) log C = 0 and Vr k 1,VXi,...i E V:o T-]Q, StlPz6xi .... illf«6xq .... Define temarl (t). n--i exp(Ej=0 ¢(T x)) n-1 If lar(T.b) - 0 as n -- ec,, which implies COlltilmity of ¢ iii symbolic distance, then çb satisfies the \VBV property. Hence if (T, X.Q) is a subshift of filfite type, then anv contilmous ftlllCtiOllS satis[v the WBV property and if (T. X. Q) is a countable Markov shift, then any unifornflv continuous functions with l'oq(T, ¢) < satisfv the WBV property ([7]). Let be the «-algebra of Bord sers of the COlnpact space X. Definition ([17], [18], [20]). A proba.bility lneasure p on (X,) is called a weak Gibbs measure for a fllnction ¢ with a constant P if there exists a sequence {K }>0 of positive numbers with lim+(1/n)logK = 0 mlch tha.t -a.e.,r, ç2' < (X'" 0")) çn, where Xi...i. (a') denotes the cylinder containing m. l'a,'n(T, 0) := sup sup le(x) - 0(g)l- }--- ,n--1 eVj=o T-J(Q) ac,yEY COUNTABLE TO ONE MARKOV SYSTEMS 2953 For a fimçtiou b X - OE, we define al opertor £ iy If satisfies l'ar(¢) + 0 (, + ), II£lll := sup.x £4l(.r) < and (04) {t'i}iG I i tll equi-cmlimous familv of partially defilwd mfiformlv «onlilut- O/lS then £ç preserves C(X) (i.e., £ " C(X) + C(X)) and is callcd thc Ruelle- Perrot-Frobenius operator. Ve rcmark that (04) is valid if V=o T-2(Q)} + 0 as . + recall the uexl result, whi«h follows rioto Theorem 5. l in [17] and Propositiou . i [18]. Len,ma 2 ([17], [18]). Let (T.X.ç) be a t,'a,,.sitiee FRS Ma,'t'oe intX G . and let be a potential qf IVBI'. Assume tlmt lh«re e.rist p > 0 and a Borel pmbabilit9 mea.sure u ot (X. U) sati.sf9inq £u = pp. whcre £* is the dual of . £. Thetz e i.s a weak Gibbs mca.ure .for with -logp. Definition. say lhat a Borel probalfility mcasure u on X is at f-conformal measuzv if d(uT)] , In order to show the weak Gibhs prol)erty of oe. we use thc [ollowing formula of thc local .]acobians with respect t() oe dx, Thus for the existence of weak Cil»bs measurcs, it is enough to show the existence of conformal measures (see 3). Lemma 3 (Theorem 2.2 iu [18]). Let u be a 'w«ak Gibbs measure for O u,«th -P. If there e:eist.s a T-ittvariant ergodic pmbabilit9meastte i t equivalent fo u with In particular, if the constant Pis the measme-theorctical pressure, thon the existence of a T-invariant ergodi« probability measure i t equivalcut to thc weak Gibbs measure u for ¢ with -P implies thc existence of a« cquilibrium state for ç (see 4). Iu order to achievc both coustructions of conformal measures and equilibrium states, we need to introduce new dcrivcd systems which are callcd jtmp transformatios ([13]). Let B C X be a uniou of cylinders of rank 1 of whk'h iudcx i belongs to a sui»set. J of I, and let Dt := Bï. Dcfinc a function R X NU {} by R(.) = inf{n 0 T G Ba } + 1 and for each « > 1. dcfine inductively = Y T.- ç*3" TR()3 ". X denote X* :=, (U=o (N.>o{R(-r) > "})) and r := U {(,....) < r'- x,,.... ç ,,}. n>l Then it is easv to see that (T*, X*. Q* = {XL}6I. ) is an FRS Markov svstem. For = =o 5T (.r). 2954 M YURI Definition. We say that an FRS Markov system satisfies local exponential insta- bility with respect to B if (05) " B 0 < 3'* < 1, 30 < F* < oc Sllch that Vn >_ 1. n--1 er\", T'(n)=sup{diam}'l}'e V T*-'(Q*)} <_r*3' *n. j=0 Definition. We say that a potential çb X - IR satisfies local bounded distorsion with respect to B if there exists 0 > 0 such that ((16) Vz_ = (iL .. il*_l) E I*, 0 < Le(/) < oo satisfying Ici(ri(x))-d?(vi_(y))l <_ Le(i_)d(x,Y) ° (Vx.y e Tli-'Xi_) and sup Z L¢(ij+...ilgl) < /I* 3= 0 Under the conditions (05-06), we can easily verify that {@*v I*} is an equi-H6hlr contim,ous fmnily (cri [19], [20]) and % Var(T*,ç*) < . Both conditions (05-06) can be eilv verified for all higher-dimensional examples in 8. In the test of this section we shall state relations between jump transformations associat.ed to B and induced maps over B. Let Rs " B U {} be the first return function defined bv Rs(a) = inf{n k 1 Tn G B}. Then we define the induced mapTB over {xG B " Rs(w) < oe} by Tsx = Tn()z and the induced Rs ()- Th(a)" Then potential B, " {x B RB, (x < oe} N by çBx (X) = we can immediately see the following facts. +(0-¢oT,), Lemma 15 (Lemma 4.1 in [18]). Suppose that B consists #fall cylinders. Then for - isaTt3- any T-invariant probability measure m witb m(B) > O, mB .-- m(Bx) invariant probability measure and m* := mBT[[3 is a T*-im,ariant probability measure, m can be written in terres of m* by Schweiger's formula (see (3) in §5) and in terres of mt3 by Kac's formula (see Lemma 16). §2. TOPOLOGICAL PRESSURE FOR POTENTIALS OF ,VEAK BOUNDED VARIATION Let (T, X, Q) be a transitive FRS Markov system and let çb : X -- ]R be a potential of WBV. For each n > 0 and for each U L/ we define the following partition functions : i:li_]=n,int(TX,, ):UDintX, x v£x=xcl[intX£) n--1 exp[ Z (pTh(x)], h=0 z,,(u, ¢) = n--1 E sup exp[ Z dpTh(x)] i:[i_[=n,int(TX, )=UDintX, xX_ h=0 COUNTABLE TO ONE MAIRKOV SYSTEMS 2955 and z(u, ¢) = We further define z,, (,):= i_:li_l=n,int( T X ,, )=UDit X q inf exp[--" 6Th(x)]. xEXi_ h=0 n-1 Z Z exp[Z 6Th(x)]" i_:li[=.int(TX,n)DintX, vi_x=xEcl(intX_) h=0 We shall define the topological pressure as thc asymptotic growth rates of these partition flmctions. Theorem 1 (%)pological pressm'e for I)otcntials of WBV). Let (T,X.Q) be a transitive FRS Markov system and let be a potential qf Il'BI2 For each U G , lina L log(U, ), lim log ,(l r, ), lim L log Z(U, ) exist and do hot depevd o r. Furtheore, the limits coincide with lim L logZ(). 1 log Z,,() the topolo9ical pvssure for call the limit Ptp(T. ) := lim,.oe . The next fa«t tan t)e verifie(l easilv. Lelnma 6. Uvder the next conditiom Z() coincides with the us'ad partition (1) For Zo X,...i, with Txo = xo, either xo intXi...i or .r0 clXj...j .for(j...j) ¢ (i...i). Let ç be thc filfite disjoint partition gcnerated by . shouM claire that if a periodic poiut x0 with period , is COlltaincd in a cyliudcr Xi...i, satisviug Xi...i, C itl for SOlne 1 G ç, then '0 OXi...i, If hot, we have a contradiction to Xo intl because of xo T(OX...i,) = O(TnXq...i). By usiug this fact. we will see that all higher-dimensional exanlples in 8 satisfy (1). The Artin-Mazur- Ruelle zeta function @.e(z) is defined by @,e(z) = exp[, Z()]. Then the radius of convergeuce of Çr, e(z) is giveu by pe = exp[limsup logZ()] -. ç define W(T) := {: X I satisfies WBV and Ptv(T,) < OE} and WB(T) := {6 e W(T) [ l'ar,6 --, 0 (,, --, 0),11611 := We can easily see that the pressure fulmtion Ptop(T, .) : W(T) --, 1I{ satisfies conti- lmity, convexity aud V6, 62 c W(T), Ptop(T, cb + 62) < Ptop(T. 6) + Ptop(T. 62)- Furtherrnore. by applying Theorem 2.4 in [7] we have the follov«ing fact. Theorem 2. î;B(T) is a Banach space and Ptop(T, .) : î;B(T) -- is a Lipschitz continuous convex function. Definition. If an FRS Markov system (T, X. Q) satisfies that VU c L/, BXi Q such that Xi C U and T(intXi) = intX, then (T, X, Q) is called a strongly transitive FRS Markov system. Theorem 3 (Topological pressure and the spectral radius). Let (T,X,Q) be a strongly transitive FRS Markov system satisfying Lt f 12 ¢ . Let ç be a potential of weak bounded variation. Then VU lg f3 l? and Vx U, linloe 1 logL;lu(x) = Ptop(T, 6). Futhermore, lim,> 1_, log IIL;ël][ = lim_oe L 2956 M. YU1RI \Ve can easilv verify ail conditions iii Theorem 3 for exalnples in §8. §3. THE CONSTRUCTION OF WEAK GIBBS CONFORblAL blEASU1RES Let (T. X. Q) lin a transitive FRS Markov svsteln and let G (T). Suppose that there exists a union of fldl cvlinders B(C X) with respect to which (T. X. Q) satisfies local eXpOlential ilstabi}ity alld satisfies local bounded disçortion. For pR(g)--I T h (£) We define the derived potential *(,r) = h.=0 (]..4)Çl*n:it('F * Y )Dit., v]...nX=XÇcl{intXL..4n ) Then l»y Theoreln 1. Smlmmble variati(ms of * allow OlW to show that 1 i,ll - g Z,,(*) := Pp(Y*,*) e (- l- Theorem 4 (A constru('lioll of confornlal lneasures via jmnl) transformations). Let (T, X. Q) bc z hmtsilite FR+" M«u'kof .sy8tem ittd lel G }V(T). Suppose that there «.rist. « utiot «t( full c,lliuder., B(C X) u,ith re.spect o u,hich (T,X.Q) satisfies local e.rpotetttiM in.l«bilil9 ami .ç«tti,sfies loco,[ boutded dislortion. .4s'zlte further lhat ]{£»*-nmin{a.&p(T*,ç*)} 1 Il < . T»«,, lhere emi.st.s a Borel probabilil m«asure on X suppovled o X* .s.o, li.sfyitg As we bave amtounced in 0. for constructing a xxeak Gibbs measure for of XVBV. we shall considcr thc following generalized Bowen's equation: The existence of a zero of the equalion (GBE) %llows Kom the standard argmnent in the case when 0 Ptop(T*,¢*) < because of contimfitv of the function should notice that the uniqueness of thc zero of (GBE) follows Kom the "'strictly'" decreasing property of the fmtction, s Ptop(T*, *-.sR) in the standard situation. Here we have no evidence of it although the funclion is decreasing. If Ptop(T*, *) < 0. then mder the assumption Ptop(T*, * - RPtop(T*. *)) < we see that Ptop(Y*, O* -- Ptop(*, *)) k Ptop(T*, * - Aop(T*, *)) = 0 and so we can reduce to the previous case. If Ptop(T*. ¢*) = . then we cannot use the st.andard argulnent. Now we corne to state the next kev lemma, which allows one to establish Theorem . Lelnma 7 {The existence of a zero of (GBE)). (i) /f 0 _< Ptop(T*, çb*) < oc,, then Ptop(T, ) > 0 a,,d s0 _> 0 satisfgi,9 Ptop(T*, * - s0R) = 0. (ii) ff Ptop(T*, çb*) < 0 o,td Ptop(T*, çb* -/ï'Ptop(T*, çb*)) < oe, then Ptop(T, çb - Ptop(T*, çb*)) _> 11 a,,d s0 _> 0 sa, tisfgi,,9 Ptop(T*, (çb - Ptop(T*, çb*))* - s0/ï' ) = 0. (iii) /f sup{s G IR: Ptop(T*,c)* - sR) = oc} = lnill{O. Ptop(T*,O*)}, then s0 _> l,lil,{(), Ptop(T*. d)*)} .sttc[t ll-at Ptop(T*. d)* - s0) = 0. COUNTABLE TO ONE MARtxOV SYSTEMS 2957 We recall the formal power serics (T,4(z) = exp[,= 1 -g- ,(¢)] which is called the Artil>Mazur-l:hmlle zeta fimction. The next product formula of zeta functions plays an important foie in proving Lenuna 7. Proposition 1 (cf. [17]). II'e tan write (2) çr,¢(e×p(-s)) = çr-,«-R(1) × Corollary 1. /.f s > Ptop(T, 0), then Ptop(T*, O* -- .s/-¢) <_ 0. Bv Theorelu 3, the assumption ]]£O*-Rmin{0,Ptop(T*,O*)}l]] < C implies either 0 < Ptop(T*, *) < oc or Ptop(T*, * - r¢Ptop(T*, *)) < oo is satisfied. He,lce it follows froln Lemma 7 that Ss0 > nlin{0, Ptop(T*, *)} satisfying Ptop(ff*, * - a'0H) = 0 and II:«,olll < . Since Q* = {x,_}i¢** consists of full cylinders and sulnlnahility of variations ,__ Iar,(T*. * - s0/i') < oo is valid, we Call apply P. Walter's argmnent in [14] to show the existence (,f ail exp[soR - ¢*]-conforlnal measure with resl)ect to T*. Lelnma 8. There exisls a Bmvl probability mea.'nw oe on X solisfging £.* ¢._soRlY : and v(intX*) = 1. In §9 we shall show the existence of an exp[s0- ¢]-conformal measure for thc zero s0 of (GBE) by using the conformal measure oe on X* and show s0 = Ptop( T, which implies uniqueness of the zero of (GBE). At the end of this section, we shall consider the case when I]£¢._tni{o.e,o(r.,¢.)}lll = oc. B.v Tlmorem 3. if * n Ptop(T , ¢*-&,]ï') = 0. then therc exists suflïciently large n such that ]] (¢-,o)* 11] = ]]£(¢_,)11] < oc, where --1 (0- So) := Z (- s°)*T* = n-1 We shall introduce a new stopping time (depending on n > 1) defined Oll X* bv R,(x) := inf{k _> n I Xi...i(x) n--1 r--I V r*-Q*} = 2 Then a new jump transformation S* detiued bv S*(x) := 7"R(X)(x) is equal to T *' and the next facts can be verified easily. Now we shall consider a two-parameter falnily of functions {(¢ - s) I (s, n) ]R x N} and the equations Ptop(T *n, (¢ - 8)z) = 0. Applying Theorem 4 gives the next result. Proposition 2. Suppose that ail conditions in Theorem 3 are satisfied. If there exist .s0 ]R and no N such that Vn _> n0, Ptop(T *n, (¢ -- S0)z) = 0. then there exists a Borel probabilitg mcasure oe o X supported on X* satisfyin9 -4-T[X, = exp[s0- ¢](Vi e I) ad p(UiFio.\i)= o. Furtherrnore. so = Ptop(T,¢). 2958 M. YURI §4. INDIFFERENT PERIODIC POINTS AND NON-G1BBSIANNESS Let (T, X, Q) be a transitive FRS Markov system and let E kV(T). The next lemma follows from the definition of Ptop(T, ) directly. 1 q--I Lemma 10. Ptop(T, ¢) >_ h=o dpTh(xo)( VxO X, Tqxo = Definition..r0 is called a generalized indifferent periodic point with period q with 1 q--1 respect, to if Ptop(T, ç) = h=0 (pTh(x0) If x0 is hot indifferent, then we calt fo a generalized repelling periodic point. If a potential of \VBV admits a generalized indifferent periodic point, then we can observe interesting statistical phenomena. More specifically, if there exists an exp[Ptop(T, )- ¢]-conformal measure u, then t he above definit ions can be described in terres of the local .]acobians with respect to u, that is, d(uT q) q--I Ix,...,q(o)(xo) = exp[qPtop(T, dp) - Z cpTh(x°)] = 1. h=0 Then we have the following facts. Proposition 3. Let .r0 be a generalized indifferent periodic point with period q with respect to c G kV(T). Let u b« an exp[Ptop(T,b)- O]-conformal m«asure. Th«n (i) Vs 1. Ptop(T, s0) = sPtop(, 0) and Vs < 1. Ptop(. sO) sPtop(T. ). (il) u(Xi...i, (x0)) decaBs subeonentiall9 fast. Proof. Bv Lemlna 10, we have Ptop(T, sdp) > s Eï dpTi(xo). In particular, if x0 is a generalized indifferent periodic point for b, then Ptop(T, sdp) >_ sPtop(T, dp). We recall that by Lemma 2 the conforlnat measure u is a weak Gibbs measure for b of WBV. Then we bave for s > 1. 1 1 where both C, and Kn satisfv tim,_oe -1 ]ogC, = 0 and lilnn--,ec_1 logK, = 0. Since lilnn--,oe _1]ogCnIÇn = 0, we bave Ptop(T. sdp) _< sPtop(V, çb) for s _> 1. (ii) fotlows frolll Proposit.ion 6.1 iii [21]. [] Let us recall that u was obtained by constructing a jump transformation in Theorem 4. Then we can associa.te the generalized indifferent periodic points to the lnarginal sets Ç>_0 D. Proposition 4. Let xo be a generalized indifferent periodic point with period q with respect to dp kV(T). (i) (Failure of bounded distortion) Cnq(XO) :-- sup exp[-ïq-l )Th(x)] - oc nq--1 C'Xil ..... q(XO' exp[Eh_0 (bTh(/)] monotonicall9 as n --* (il) xo Ç),>0 D,,. COUNTABLE TO ONE MARKOV SYSTEMS 2959 Pro@ Since C,(Xo) is the distortion of d(vT') d over cylinders Xil...i(xo), (i) follows from Lemma 6.1 in [211. S,,ppose xo ¢ 0ao D. Then by Sublem,na A (see 9) we bave .ro X*. Since Vr,(T*, ç*) < oe ilnplies that C,q(Xo) cannot increase monotonically, we have a contradiction to (i). complete the proof. Remark (C). We claire that 0 D, can contain repelling periodic points. If we have a T-invariant probability mea.sure p equivalent to u via Kac's formula (Lemma 16) or Schweiger's formula (3) in 5, then the invariant densities d/du are typically mfl)ounded at indifferent periodic poinIs with respect to u (Lemma 6.2 in [21]) so that we tan sec interesting phenomena from a statistical point of view ([19], [21]). For example, under the existence of a generalized indifferent periodic point ,r0 with respect to , Ihe raie of decay of correlation may be slower than u(X..., (x0)), which decays subexonentially fast by (il) in Proposition 3. referee [21] for fllrther details. On the other hand, the Dirae measure m supported on the generalized indifferent periodic orbit with respect o satisfies Ptop(T, ) = h.(T) + f. çdm. Hen«e if we can esIablish a variaIional principle for the topological pressure and tan construct a T-invariant measm'e p equivalent to Ihe weak Gibbs lneasure u for wiIh -Ptop(T,), theu by Lemlna 3 we sec immediately failure of uniqueness of equilibriuln states. Furtherlnore, by the definiIion of indifferency we can show faihlre of Gibbsimmess of equlibrium states for ç with generalized indifférent periodic points. Theorem 5 (Characterization of non-Gibbsianness). Suppose that a potential ç with Ptop(T, ) < admits a generalized indifferent periodic point x0. Then there is no Borel probabilit meas'ure that is Gibbs for . §5. EQUILIBRIUM STATES FOR POTENTIALS OF 'VEAK BOUNDED VARIATION Let (T. X, Q) be a transitive FIlS Markov svstem and let ]lIT(X) be the set of all T-invariant probability measures on (X,,T). For m E MT(X), I,n denot.es the conditional information of Q with respect to T-¢ -. We denote JIIT(X,c) := {m E MT(X) I I + 4) L(m),either h,(T) < oc or .,,.O dru > -ec is satisfied}. Theorem 6. Let (T X, Q) be a transitive FRS ]llarkov systern ad let dp W(T). Suppose that there ezists a union of full cylinders B(C X) with respect to which (T, X, Q) satisfies local ezponential instability and c satisfies local bounded distof tion. Let u be the exp[Ptop(T,¢5) - dp]-conforrnal measure supported on X*. As- sume further that F := Ç),>0 D, consists of periodic points. If fx* Rdu < oc and H,(Q*) < oc, then there ezists a T-invariant ergodic probability rneasu tt equivalent to u that satisfies the following variationai principle: dp) = bu(T) + .. dp dp = sup{ h,(T) + \. dp dru I m ]lIT(X, dp) is ergodi«}. Ptop (T, If ET(X, cP) := {m MT(X, cp) [ h(T) + fxcpdm = Ptop(T,¢)} contains at least two elements, then it iInplies ph.vsically coexistence of different phases, which is so-called "phase transition". Phase transition may be related to failure of Ihe Gibbs property of equilibrium states (see Theorem 5). 29ô0 M. YURI Corollary 2 (Phase transition). i'e assume all conditions in Theorera 6. ff F consists of 9eneralized indifferent periodie points with respect to ch, then tbe set of equilibrium states for ch is the convex hnll of p and tbe set of iTvariant Borel probabilil9 meas,u«'es supported ou F. In order to prove Theorem 6. we need a sequence of lemmas. Let 3IT*(X*) denote the set of ail Borel probability measures on X* invariant under T*. For ch* - sR we define Mr.(X*,ch* - sR):= {m* G MT.(X*) [ either b,.(T*) < or i.(çb* - sR)dm* > -: is satisfied}. Le s0 = Ptop(T,¢). Then P. Walter's lnethod in [14] can apply for T* and for * - soRso that there exists the unique equlibrium state ff* equivalent to u and the following variatiolm.l principh, is valid: 0 = r(T*, O* - SoR) = h,,. (T*) + f (* - = ,p{,,,* t.(x*,ç* -.t) I ,«(T') + [ (« s0R)dm*}. « Since ,, I "ar. (T «, 0* - 0R) < OE implies the i,ounded distortion property with respeçt to u Slip Slip slip d(eT.n ) < -- t Vj=o T*-(Q* d we can show ergodicity and Bowen's Gibbs property for p*. If fx* dp* < OE, then the next Schweiger's forlmfla ([13]) gives a T-invariant ergodic probability mesure p equivalent to oe that satisfies p(B) = (x-- dP*) - > 0" and bv Lemma 5 for f ¢ Ll(p), [ Ji\-. -î--(g)- fTi(x) d#* fB, --î=d ()- (3)* fdlt = fx. @* = f, ,d, (cf. Lemma 4.2 in [18]). Sin«e fç. Rdp* < gives the equality and Hu. (Q*) < OE, gives b.(T) < ,, we can establish the following characterization of the zero so of (GBE). Lemma 11. /fp* e Mr- (X*) is ergodic and satisfies ht. (T*) +f\.. (* - «oR)dp = O, fx. Rdp* < cx and Ht.(Q* ) < cx, then S 0 ,..(T*) + L\-. O*dp* f\.. Rdp* = h t, (T) + f\ chdp, where p is obtained bg fornrula (3). COUNTABLE TO ONE MAFIKOV SYSTEMS 2961 By Lemma 11 we have a T-invariant ergodic probability measure # equivalent to that satisfics Ptp(T, ç) = b,(T) + fx çdl*. Lemlna 12 (Lelmna 4.4 hl [18]). g a T-invariant probabilitg measure m sotisfies re(B1) = 0, then F := n0 Dn is a full measure set with respect to m. Proof of Teorem 6. Bv Lelnma 11 for ail T-ilvalialt ergodic probability lneasures m on X with re(B1) > 0 and m Iç(X. ¢), we can establish ,(T) + £x- Ca,.- P(T. ) > 0= /I(B1) -- (B1 ) On the olher hand, bv Lemma 12, any T-inval-iant ergodic probability ineasure m on X with m(B) = 0 satisfies re(F) = 1. In particular, if F := ,,0 D, consists of periodic points, then ,(T) + fx dm = J çdm Pwp(T, ç), which completes the proof of Theoreln 6. §6. TtlE ('ONSTRUCTION OF O-FINITE ('ONFORMAL MEASURES VIA IND[X'ED MAPS Let (T, X. Q) be a transitive FRS Markov system and ç 6 çç(T). Suppose that (T, X. Q) satisfies local exponential instability and ç satisfies local bounded dis- tortion with respect to B1 : j Xj(ff C l). Let .4 = cl(iotXj) for j E J and put A := Uj .4. define the first return function R4 " A U {} and the induced map Ta over { A R.4() < OE}. Bv the Markov property, there exists a partition of the set B A) = {.c ¢ .4 Ra(.r) = k} for each k 1 so/bat T restricted to the interior of each element of the partition is a homeomorphisln onto its ilnage. 4 denotes the set of all indices corresponding to such elements of the partition of ,>1 Bi -A)" Then {v, : i ¢ IA } is a familv, of extensions of local inverses of . A()-- çTh(x) _ sR4(x). Bv For s Ç N &lld a'e U=I [:A) We defille *}2)(w) = h=0 Lelmna 4 we can easily see the next fact. Lemma la. If each Ai C A satisfles TAs = X the, We recall the [ol]owing resu]t in [6]. Lm a ([6]). Z I1£,111 < . t P»(«,é[2 )) «ti,,, ,,t{ e - P(4, é ")) e }. We suppose that 1 £O1]SSt8 of sing]e fui] cylhder X and the [o]lowing con- ditions are satisfied [or A = (05)* 90<< 1,0<<,such that and there exists 0 > 0 such that (06)* V = (it...iN) E IA and all 0 j < I1, 0 < Lé(i+l...ill) < satisfying le(%+,...,. (z)) - ¢(v.6+,...., (u))l L(i+ ...il)d(z,U) (W.u e A) 2962 M. YURI alld I1- sup E L4)(iJ+"'ili-I) < ilA Since the conditions (05-06)* allow us to establish the WBV propertv, of WA a'(»), by Theorem 1, lim, 1 log Zn(¢}}):= Ptop(T4 , CA(»})' where n( »'): X X X " " I n -!n x:x Furhermore, (05-06)* gua.ramee equi-Halder comhmity of {)q : ln} and E% '-(T, ) < - " i lG,il < . th Gç», .C(A) C(A). Theorem 7 (A construction of -finite conformal measures via induced maps). Let (T X Q) be a transitive FRS Markov sgstem and let (T). Suppose tbat th«r« «xts a full cylind«r Xj e Q satis]ying (05)* and (06)* for A := cl(itA3). min{O.Ptoç(TA,OA II the (i) 3so e with Ptop(4,¢ s°)) : 0 (a g«n«ralized Bowen} «quation); (ii) there eists a Borel probability measure oeA on (A. A) with oeA({X A RA(X) < }) = 1 satisfying E(»oUA = (iii) there exi.sts a a-finite measure u on X satisfyin9 £u = [expso]u and (iv) in particular, if u is finite, the. Ptp(T. ¢) = so. Proof of Theorem 7. B.v Lelnmas 13-14 and Proposition 1, we have the existence of s E IR for which Ptop(TA, ¢ )) = 0 and (d adlnits an eigenvalue 1. Then we can apply the main theorem in [5] so that (i)-(iv) are obtained. [] The next result gives a criterion of finiteness of u. Proposition 5 (A criterion of finiteness of e). Suppose that all assumptions in Th«or«m 7 arc satisfi«d. Thon ee(X) : fA exp[s0- ¢]deeA + 1. In particular, if infxeA ¢(x) > --oc, then u is finite. Proof of Proposition 5. Let I' = {/ [Xt C D N } First we note the following for- mula of u, which was obtained iii [5]: Then we see that oe(X) is equal to k k=l j_eIA.]jl=k+l v_(A) /=1 COUNTABLE TO ONE MARKOV SYSTEMS 2963 because of the fact t.hat Xi C TA(V/ I'). By conformality of V A this coincides with &:l jI.«,lJl:c÷l L (A) k exp[( Z 0T'- s0)(x)] exp[--0(A°)(z)]&,A(Z) + 1. §ï'. THE CONSTRUCTION OF MUTUALLY SINGULAR EQUILIBRIUM STATES In this section, we show the existence of mutually singular non-atomic equilib- rium states by using induced systems. Lenmla 15. Let (T, X. Q) be a transitive FRS Markov system and let ¢ C V(T). Suppose that there exists a sequence of full cylinder:s { " M Xi}i=I(M _< oe) that sati$fi«s (05)* and (06)*, infeA ' ¢(x) > -oc, and ffgA rnin{O'Pt°p('TAi for each Ai --- cl(bdX). Let Fo := X and for each i > 0 define indu.ctively Fi+I --- Ç)n=o T-n(F fq AC+l)(C F). [l'e assume that for each i > O. dF iCA,+I for the Borel prvbability measure 'r,A,+ on Fi Ai+I obtained in Theorm 7. If M = F := F con.ists of periodic points, then ther exists a T-ivariant eryodic prvbability measure p equivalent to an exp[Ptp(T, ¢) - ¢]-conformal measure that satisfies the following variational principle: Ptp(T. ¢) = h(T) + ,. ¢dp = sup{h(T) + f,. d,n , m Mr(X, ¢) is eryodic}. The equilibrium state p for ¢ is not necessarily unique. Theorem 8 (Phase transition and singular equilibrium states). IVe assume all conditions -in Lemma 15. If F := I___/1 Fi consists of generalized indifferent periodic points with respect fo ¢, then there exists a sequence of ergodic equilibrium states {#i IM that are mutually singular and the set of equilibrium states for ¢is the J convex hull of {#}îl and the set of invariant Bord probability measures support«d on . Lemma 16 (Kac's formula). If f A lAd(oe[A ) < oe and PA is a TA-invariant ergodic probability measure equivalent fo oelA, then the next formula gives a T-invariant ergodic probability measure # equivalent to , RA (z)-- 1 #(E)/#(A) =/A Z i=0 lE o Ti(x)d#A(X)(VE e Lernma 17 (Finite entropy condition). Suppose that for si := Ptop(TIr,_, r._,A,d%_,A, > --oe. Then H,r.«,_ ' (QrA,+,) < 2964 M. YU1RI Proof of Lemma 17. Let A = Fi ç) Ai+l and s = si. Since we have V3_ E I4. eA(Xj)_ =/a exp[¢})(vj(x))]deA(X)" - -- > exp[--s]j]_ evA)inf exp[¢)(x)], the bounded distortion for a(°) allows us to see that EIA z - 14 OE IA whcrc C is thc bomdcd distortion constant exp[D(1 -)-l(dimnX)°]. These incqualitics allow ont to cstab}ish H,,A ((A) [--¢A )dt'.4 + log C + s H 4duc < OE. IA Z ProofofLcmma 15. Since n__ Var,(T.4 ,v'A < is satisfied, it follows from Theorem 7 that A, on At satisfying £(s)FAl = F.41 for S = Ptop(T,) and A 1 an exp[Ptop(T,) -]-conformal measure on X. Fnrthermore. bv Proposition 5 we see that U(Al) -- PAl" The bounded distortion allows Olm to obtain an ergodic TAI- invariant probability mesure PAl PAl with a density dlt 41/doeAl awav froln zero (s) and infinitv. Furthermore, by [14] there exists an equilibrium state PAl for with respect to T4 that is ergodic. In particular, since H,A (Q_41) < we have for 81 : Ptop(T, ), top(TAl, CÀï )) 0 l'"Al (1) + ¢81 )d'', ) hA, («1) + 1 1 for all a,- invariant probabilitv, measures mA1 e MT, (A, ¢(s))A, (cri [14]). These inequalities and Lcmma 16 allow us to bave a T-invariant ergodic probability mea- sure g oe that satisfies lq(A) > 0. pi(F1) = 0 and (**) 0 = t,(A,)-(. (T) + .. (¢- 81)dl) for ail T-invariant ergodic probability measures m 3IT(X. ¢) satisfying re(A1) 0. (**) is equivalent to the inequalities .s = h m (T)+ f.x- dp h(T)+ f x- dm. On the other hand. any m e MT(X,¢) satisfying t(.4) = 0 is supported on F. In fact, since X = (Ui=0 T-A,) u (0(T-A)«). re(A,) = 0 and T-invariance of m give m(F) = 1. Thus the set of ail T-invariant probability measures m supported on F coincides with the set of ail T-invariant probability measures m with m(A) = 0. Since (Tr,,r. Qrl := QF) is a subsystem of (T,X. Q). we can apply the above arguments for the induced svstem (%nr,, A2 F. Q%nr, )- That eX rn--1 is, for s = Ptp(TIr,,¢ ) := lim log :lfl= ,=.æer, "Pth=O CT(x)] and for the associated potential WA:r, (x) = h=0 - . COUNTABLE TO ONE MARKOV SYSTEMS 2965 our assmnptions allow us to establish the i»ounded distortion for '.4cr so that there exists a 4rl-invariant ergodic probability measure PAr that satisfies > h« («r) + CA 2Fl for ail 4=r,- invariant ergodic probabi]ity ineasmes m.4=F ITn=r (A2 FI, ç(s=) ). Let p be the T-invariant ergodic prolmbilitv measure supported AF arising ff'oto It.4p via Kac's formula. Then sz = ht,=(T) + f x dp2 h(T) + fx çdm for ail T-invariant ergodic InObability measures m Mv(A. ¢) supported on F that satisfy m(F 5 4z) > 0. Inductively we have a decreasing sequence {s}, where s = Pp(Tr_,ç) and a sequ«nce of T-invariant ergodic proba- bilitv measures {Yi al . }i= such that pi is supportcd on 1" _l.tt,(I'i_ A,) > 0 and probability measures m G -lr(X,ç) supported on l'i-1 with m(I'i_ Ai) > O. Since yi(Fi-1) I and pi(f/) 0. = = {11'}i=1 arc nmtually singular. Filmlly. for every T-invarialt measure supported on F that consists of periodic points we bave s 2 ,,,(T) + ./v çd,,,. Since {s}e is decreaing, we complete the proof. 8. APPliCATiONS In this section, we show some examples of transitive FRS Markov svstems to which our theorems 1-8 can apply. Exanple 1 (Brun's map [lai, []8], [0]). ter X = {(.,',,.r) e "0 < 1}. and let for i = 0.1, 2 where we put = 1 and z = 0. T is defined bv T(Xl,X2) = ( z, z= ) Oll 1 T(,FI,3.2) : (x2 1 1) on Then Q = {X,}_ 0 is a Bernoulli partition and (0.0) is an indifferent fixed l)oint (i.e., det DT(O. 0) = 1). Since T is a continuous piecewise C 2 nmp and a(n) = n -1, all conditions (01)-(04) are satisfied and dynamical instal)ility is polylmmial. e that ç = -logdet DT is i)iecewise Lii)schitz continuous so that ç is a potential of XVBV. Furthermore, since each periodic point is contained in a single cylinder the property (1) is satisfied. Define B = X1 U X2. Then T* satisfies the uniformly expanding property and a direct calculation allows us to establish (06) for ç = -log[det DT[ (see [18] for more details). Hençe we can apply Theorems 1-8. In particular, we can see that Ptp(T, ç) = Ptp(T*, ç*) = O. Example 2 (InhomogelmOUS Diophantine approximations [13], [15], [16], [17], [19], [20], [21]). For the transfornmtion defined in the introduction (Example A), we can directly verify all conditions (01)-(04). In fact, we can introduce an index set I={(î) a.bZ,o>b>O, ora<b<0} and a partition {X()" ()I}, where X()= {(a',g) X" a = []- 2966 M YURI we can veri-IÇ(b) _< log(1 +a( -2)) and a(n)= O(n -1) (see [15], [20]) (cf. [16], [19], [21]). Hen«e b = -logldet DTI is a potential of WBV. Since each periodic point is contained in a single cylinder, the property (1) in Lemma 6 is satisfied. Let D, be the union of cylinders of rank n containing indifferent periodic points and let B,, = D,_D,. Thei1 the junlp transformation T* Ui= Bi X defined by T*(x, ) = Ti(x, ) for (x,9) B, satisfies exponential decay of dialneter of «ylinders (see [19]), and we can veriÇv the validity of (06) for ç = -log[ det DT[. Indeed, for G I* with 1] = , L¢() 3/ 2 and so Il- oe 3 EI* j=O n=l Hence we hae smmnability of Var,(T*, *). which allows us to apply Theorems 1-8. Example 3 (A complex continued fraction [51, [12], [15], [22]). For the transfor- mation T, dcfined in the introduction (Example B), we define .k'«+ = {z G X [1/z] = ,a +'m} for each ,, + m G I := {ma + n" (m.) G Z 2 - (0,0)}. Then we bave a countable partition Q = {X},e of X that is a topologically mix- ing Markov partition and satisfies (01)-(03). The inverse brmehes to T take the forin vj(z) = 1/(j + z), where j G I and the vj satisfy (04). Therefore the inverse branches of the nth iterate of the transforllmtion T" take the forin ,t 1 Ujl ..... in(Z)- p" + zp.-- and Ivj, ..... j.(z)l = q + zq-i ]q + zq-] where p = jp_ + Pn- and q = jq_ + qn-, n 1. and p_ = a, P0 = 0 = q_ and q0 = a. If the string j .... ,j_ corresponds to a cvlinder that contains one of the indifferent points, but the longer string j ..... j corresponds to a sub-cylinder disjoint from the indifferent periodic points, then ujl,...,j n is an imerse branch of the jmnp transformation T* which is uniformly expanding. For ¢(z) = -log [T'(z)], WBV and (06) are satisfied. Further details tan be found in [22] in which multifractal formalism was established by applying our Theorems 1-8. §9. APPENDIX- PROOFS OF RESULTS IN §§2 AND 3 For the proof of Theorem 1, we first verify the following facts. Lenmla 18. (18-1) VUk E bl, Zn(Uk, c) > 0 for all n > S. (18-2) VU ,Vn,m > S.+(U,) (U,)(U,). (S.ubadditivity) (18-3) VU.Ut U. z+»,+» (u, ) z(u,é)(c»,c») s»--I st --1 Proof of Lemma 18. (18-1) follows kom Lemma 1 and emark (A) (18-2) %llows flore the definition of (E) directly. Since +s,+s(G,) is bounded flore below bv inf exp[ zXt :l[=s.int(TX,, )=U .itXi 1 CUl COUNTABLE TO ONE MARKOV SYSTEMS 2967 n-1 x Z inf exp[ Z çflTh(x)] x¢Xj_ j:lj l=n,int( T X j )=Uk Dint X 1 h=(J X Z xXt t_:ltl=s ,int(TXt» )=U ,intXt I CU and the transitivity condition allows one to establish si--1 inf exp[ Z h=0 and (18-3) follows from the WBV property of ¢. Proof of Theorem 1. By (18-2) in Lelmna 18 a.nd the BV property of ¢, both lim ± log(U,¢) and lim - logZ(U,¢) exist for each U b/. Since -- 1 Zn(U, ¢) _< Zn(U, ¢) _< Zn(U, ¢), by the WBV property of¢, lin,.._,o log Z,(U, ¢) also exists and all the limits coincide. By (18-3) in Lemlna 18, it is obvious that the limit does hot depend on U. Noting min<-_<N Zn(¢,Uj) <_ Z,(¢) <_ ;=1Z,(¢,Uj) allows one to complete the I)roof. [] In order to prove Theorem 3 we first show the next result. Lemma 19. (19-1) VV e "12 and Vx I; £,l-(x) _< _,u,_v Z(U, ¢). (19-2) Vx Uk H, L:;1u(x) > Z,(Uk,¢). (19-3) Z,(¢) >_ X()(1) Ç Q satisfles X()(1) ç U ad T(int X(1)) = int X. where Proof of Lemma 19. We first note tha.t 1T = Uxjcv Xj because of the Markov property of Q. Then for x l assertion in (19-1): the following inequalities allow us to have the jI:X)CV Utld:xUt (ji2...in):TXin=Ut h=0 Utld:UIDV (ji2...in):TXn:UIDVDX h:0 Ut L¢:Ut D V (ji...in ) :T X, =Ut DX <_ zo(v,ç). Ut lA:Ut Dt" n--1 exp[ Z h-----0 2968 M YURI For a" E Uk, we have the following iuequalit, ies, xvhich give (19-2): ;lvk(*) = jI:X CUk (ji2...in):xTX n -> exp[- CTh(vji»..,.x)] h=O h=O exp[- ¢h(vji»..i,x)] >_ Z,(Uk, ¢). h=O By the WBV property of ¢, Z(¢) is |)ollnded fl'oln below bv Then (19-3) ff»llows frolll the Markov property and the strong transitivity. [] Prvof of Theorem 3. By (19-1,2) iii Lennna 19, we have for x E V C U, 1 lira sup - log £lv(x) < n --- oc 'Il lin, -1 logN( III&X n(Ul,¢)) = Ptop(T, çb) n--,oe ?l I</<N log£lu(X) > Ptop(T. çb). Out aSSUlnption Uk ff 12 in,plies U = and lira inf,_ g _ 1". Hence we bave the first assertion. The test of the assertions follow fronl (19-3) in Lenmm 19 ilnmediatel b. Iii order to prove Proposition 1. we need two sublemmas. Sublemma A (Lenuna 3.1 iii [17]). [>0 D, ad X* are positively T-invariat. Furtherv,o're, Um=l T*-m(Ç],>o D,) contains ro periodic points. Pwof. The result follows from the equality (4) R(Tx) = R(a') - 1 (R(x) > 2). Sublemma B. Deflne T'(X, T):= {x X I 3(i...i,0 I" such thatvh...i,x 7),(X*,T *) := {.c e X I 9(q ...i_,) e I *' such that v,,...iz = x}. Pro@ Since x 7),(X,T)ç X* visits B1 infinitely oflen, we Call find a point y G 7)(«\*,T *) for some I <_ n such that TJy = :r for solne j < R(y) and COUNTABLE TO ONE MAF/KOV SYSTEMS 2969 l-1 ,=0 R(T*m(Y)) = n. By the property (4) the converse is also true. Since I-1 I-1 R(T*m y)-I m=0 m=0 h=0 I--1 E=o (T*) - = (¢- h=O we have the rest of the assertion. rz=l mT)n(X,T)OX * Proof of Proposition 1. By Sublemma A we first note that 7),(X, T) = {7)n(X, T)ç fqn>0 D } t2 {P(X, T) C X* }. Then we see that (,¢(exp(-s)) is equal to çlo>o»,o(exP(--8)) exp[ exp[-ns] x exp[n-1 CTh(x)]] n=l x(X,T)X* h=O Define for > 1 > 0 E* _ ,, := {v e ç,(X*,T*) = such that R(T*'())= , and z = Then ç,(X*, T*) = U>t E*.t and ç,(X. T)X* = U<, E,,t. Bv. Sublemlna B we see t hat for z X*, .--1 h=0 n-1 We complete the proof. = ProofofLemma 7. (i) Bv Lemmas 13-14. we have continuity of the flmction Ptop(T*,¢* - sR) on int{s e IR I Ptop(T*,¢*) E IR}. Then the existence of a zero s0 > 0 of (GBE) follows from the standard argmnent. Since Ptop(T*. çb* - 80/) = 0, by Corollary 1 we have so <_ Ptop(T, ¢). If Ptop(T, ¢) < 0, then we bave a contra- diction. For (ii), replacing ¢ by ¢- Ptop(T*, ¢*) allows us to reduce t.o the case (i). (iii) Since the case 0 _< Ptop(T*, ¢*) < oc is covered by Lemma 7(i), we suppose either Ptop(T*, ¢*) = oc or Ptop(T*, ¢*) < 0. If Ptop(T*, ¢*) = o% then sup{s e IR" Ptop(T*, ¢* - sR) = oc} = 0. Hence Vs > 0, Ptop(T*, ¢* - sR) < oc. Since the function s --, Ptop(T*, çb* - s/) is decreasing and continuons on int{s IR ] Ptop(T*,¢* - sR) IR}, we have lim-m Ptop(T*,¢* - sR) = oc so that for suflïcientlly small s > 0. Ptop(T*, çb* - 8) > 0. Oll the other hand, it follows froln 2970 M. YURI Corollary 1 that Ptop(T*, ¢* - sR) < 0 for s > Ptop(T, ¢). Hence we have a zero s0 _> 0 of (GBE). le Ptop(T*,¢*) < 0. then sup{.s " Ptop(T*, ¢* - ) = } = Ptop(T*, ¢*). The sanie argmnent as those for the previous case allows us to have a zero .% k Ptop(T*, ¢*) of (GBE). d(vD = exp[* -- s0R]vi. If Proof of Theorem 4. Bv Lemma 8 we have for i G I*, a - - " d ,Z, exp[h=0 Thvi=...i.Z -- son]. Since the property (4) R(Tz) = R(m) - 1 (R(m) k 2) implies a'-. e I*, the equality du - du (ii3---iz)d(m'*=**")du now ,, to tht v& c D,, ,., = xp[O(,)- 0](V* e X*). O the other hand. we know that the above equality holds for Xi C B since i I*. Finally. we establish VX, e Q. ].x,(.r) = exp[s0- O(z)](Vz e X*). It follows from Lemma 2 and Theorem 2.1 in [18] that s0 = top(T, ). The sertion u(Ue 0X)(= u(U,e , &,(X))) = 0 follows from u(i,,lX) = 1. which is obtained bv Lemma 8. complete the proof. 1 EFERENCES 1. J. Aaronson, M. Denker and M. Urbadski, Ergodic theory for lllarkov fibred systems and parabolic rational maps, Trans. Amer. Math. Soc. 337 (1993), 495-548. MR 94g:58116 2. R. Bowen, Equilibrium states and the ergodic theory of Anosov diflfeomorphisms, Springer Lecture Notes in /Vlathematics 470. Springer, 1975. Mil 56:1364 3. M. Denker, F. Przytycki and M. Urbafiski, On the transfer operator for rational functzons on the Riemann sphere, Ergodic Theory and Dynarnical Systerns 16 (1996), 255-266. 97e:5819ï 4. M. Denker and M. Urbafiski, On the existence of conformal measures, Trans. Amer. Math. Soc. 328 (1991), 563-587. Mil 92k:58155 5. M. Denker and M. Yuri, A noie on the constructzon of nonsingular Gibbs measures, Collo- quiurn Mathernaticurn 84/85 (2000), 377-383. MR 2001k:37013 6. M. Denker and M. Yuri. Partially defined in]in, te terated ]unctional systems, In preparation. 7. D. Fiebig, U. Fiebig and NI. Yuri. Pressures and Equilibrium states for countable Markov shifts, To appear in Israel J. Malh. 8. P. H. Hanus, R. D. Mauldin and M. Urbafiski, Thermodynamic formalism and multi-fractal analysis of conformal infinite iterated functional systems, Acta Math. Hungar. 96 (2002), 27-98. 9. Il. D. Mauldin and M. Urbaflski. Parabolic iterated ]unction systems, Ergodic Theory and Dynarnical Systerns 20 (2000), 1423-1447. MR 2001m:37047 10. Omri Sarig, Thermodynamic formalism for countable Markov shifts., Ergodic Theory and Dynarnical Systerns 19 (1999), 1565-1593. M1R 2000m:37009 1 l. Y. B. Pesin, Dimension Theory in Dynamical Systems. Contemporary Views and Applica- tions, Chicago University Press, 1997. MR 99b:58003 12. M. Pollicott and M. Yuri, Zeta ]unctions for certain multi-dimensional nonhyperbolic maps, Nonlinearity 14 (2001), 1265-1278. MIl 2002h:37036 13. F. Schweiger. Ergodic theory of fibred systems and metmc number theory. Oxford University Press, Oxford, 1995. hlR 97h:11083 14. P. Walters, Invariant measures and equilibrium states for some mappings which expand dis- tances, Trans. Amer. Math. Soc. 236 (1978), 121-153. MI/ 57:6371 15. M. Yuri. On a Bernoulli property for multi-dimensional mappings with finite range structure, Tokyo J. Math 9 (1986), 457-485. MR 88d:28023 16. M. Yuri, On the convergence to equilibrium states for certain nonhyperbolic systems, Ergodic Theory and Dynarnical Systerns 17 (1997), 977-1000. MR 98f:58155 17. M. Yuri. Zeta ]unctions for certain nonhyperbolic systems and topological Markov approxi- mations. Ergodic Theory and Dvnamical Systems 18 (1998), 1589-1612. Mil 2000j:37024 COUNTABLE TO ONE MARKOV SYSTEMS 2971 18. M. Yuri, Thermodynamic formalism for certain nonhyperbolic maps, Ergodic Theory and Dynamical Systems 19 (1999), 1365-1378. lkIl 2001a:37012 19. M. Yuri. Statistical properties for nonhyperbolic maps with finite range structure, Trans. Amer. Math. Soc. 352 (2000}, 2369-2388. MP 2000j:37009 20. M. Yuri, Weak Gibbs measures for certain nonhyperbolic systems, Ergodic Theory and Dy- namical Systems 20 (2000), 1495-1518. lk|lR 2002d:37011 21. M. Yuri, On the speed of convergence fo equilibrium states for multi-dimensional maps with indifferent periodic points., Nonlincarity 15 (2002}, 429-445. MIR 2002k:37006 22. M. Yuri, Multifractal Analysis of weak Gbbs measures for Intermittent Systems., Commun. Math. Phys. 230 (2002}, 365-388. DEPARTMENT OF BUSINESS ADMINISTRATION, SAPPORO UNIVERSITY. NISHIOKA. TOYOHIRA-KU. SAPPORO 062-8520. JAPAN E-mail address: yuri0oEath.sci.hokudai.ac.jp, yuri0oEail-ext.sapporo-u.ac.jp TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2973-2989 S 0002-9947(03)03257-4 Article electronically published on March 14, 2003 STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS OF ELLIPTIC SYSTEMS D. G. DE FIGUEIREDO AND Y. H. DING ABSTRACT. We study existence and multiplicitv of solutions of the elliptic system -Au---- Hu(x,u,v) in Q, --A-v ---- -H,(x,u,v) in Q, u(x) = v(x) : 0 on where Q C N, N 3, is a smooth bounded domain and H We assume that the nonlinear term H(x, , ) I1 + v[ q + B(x, u, v) with lira R(x, u, v) _ O, whcre p Ç (1, 2*), 2" :: 2N/(N - 2), and q Ç (1, oe). So some supercritica} systems are included. Nontrivia} solutions are obtain. Yhen H(x,u,v) is even in (u,v), we show that the system possesses a sequence of solutions associated with a sequence of positive energics (resp. negative energies) going toward infinity (resp. zero) if p > 2 (resp. p < 2). Ail results are proved using variational methods. Some new critical point theorems for strongly indefinite functionals are proved. 1. INTRODUCTION AND MAIN RESULTS Cousider the following elliptic system: -Au : H(x, u,v) in . (E) -Ai, = -H.(x, , v) lu . (x) = v(x) = 0 on where C ]K N, N _> 3, is a smooth bounded domain and H x ]K 2 -- ]K is a C'- function. Here H denotes the partial derivative of H with respect to the variable u. Writing z :- (u, v), we suppose H(x, 0) --- 0 and H: (x, 0) -= 0. Then z : 0 is a trivial solution of the system. Iu this paper we discuss the existence of uontrivial solutions. Roughly speaking, we are ,nainly interested in the class of Hamiltonians H such that H(x.u,v)[u[P+[v[q+R(x,u,v) with lira R(x,u,v)=0. I Il » ÷ Il where 1 < p < 2* := 2N/(N - 2) and q > 1. The most intercsting results obtained here refer to the case when q >_ 2*, v«hich correspond to critical and supercritical problems. The case when q < 2* has been studied by Costa and Magalhfies [5], Received by the editors June 18, 2001. 2000 Mathematics Subject Classification. Primary 35J50; Secondary 58E99. Key words and phrases. Elliptic system, multiple solutions, critical point theory. ()2003 American hlathematical Societ 2974 D.G. DE FIGUEIIEDO AND Y. H. DING [61 and Benci and Rabinowitz [31. See also Bartsch and De Figueiredo [21, De Figueiredo and Magalhàes [7], De Figueiredo and Fehner [8] and Hulshof and van der Vorst [11], where similar systems also leading to strongly indefinite functionals have been studied. However, only subcritical systems have been considered in those papers. Letting 2. = 2*/(2* - 1) = 2N/(N + 2), we assume that H(x,z) satisfies the following condition: (H0) therearep (1.2"), q (1 ) andr (1, l+q/2.)suchthat, for all (., z), IH(x. u,v)l 0(1 +11 p- +[vl -1) a.nd IH,,(x,., v)l < r0(1 + lui p-1 + Ivlq-1). In ail hypotheses on 11(z, z) tho 7i's denote positive constants independent of (x, z). We note that if q < 2", thon 2. < q/(q - 1), i.e., q- 1 < q/2.. Hence, it is possible that q < -r < 1 + q/2.. However, if q > 2", then 7- < q. Furthermore, we relnark that 7- can be very large, if q is sufficiently large. In addition, we need distinct conditions on H corresponding to the ces when p>2, p<2orp=2. First. consider the ce when p > 2. In this ce, we assume the following three conditions: (H1) there are > 2, u > 1 and Rt 0 such that H(x,z)u + H.(x.z)v 2 H(x,z) whenever Izl R1, with the provision that v = g if q > 2: (H2) there are 2.(p- 1) p and 2.( - 1) < such that H(,z) OE l (ll + Il ) -- 2 O all (,z). and = q if q > 2*; (H3) H(,0, v) OE 0 and H(z,u,0)= o(11) as u 0 unifor,nly in z. ç prove the following results. Theorem 1.1. Let (Ho) be satisfied witb p > 2. If (Ht) - (H3) hold. then (E) bas at least one nontrivial solution. In order to provide some more transparent hypotheses under which the above result holds, we next present some conditions on H that are sufficient for (Ho), (Ht) and (H2) to hold: (H) there are p (1, 2*) and q (2, 3c) such that, for ail (x. z), and IH,,(x,u,v)l <_ "),0(1 + lui p-1 + Ivl §-1) IH(oe, u,v)l <_ o(1 ÷ Izl p-1 4- (H[) there are/ > 2 and RI OE 0 such that H=(x, z)u + Hv(x, z)v > ¢tH(x, z) (H) for p and q as above, H(x. z) >__ ")'1 (11 p + Ivl q) -- "r2 whenever Izl >_ for ail (x, z). STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2975 Theorem 1.1'. Let (HIc) be satisfied voith p > 2. If (H[),(H;), and (Ha) hold, then (E) bas at least one nontrivial solution. Theorem 1.2. Let (Ho) be satisfied voith p > 2. If H(x,z) is even in z and satisfies (H1) and (H2), then (E) bas a sequence (z,) of solutions with eergies I(z) := t (}(IVul -]Vvl u) - H(z,z)), going fo oe as ,, . In order to describe the other results, let «(-A) denote the set of ail eigenvalues of (-A,H()): 1 ( 2 3 "" ". We now consider the case whcn p < 2. makc the following assumptions: (Ha) there are it G (1, 2), u OE 2 and 3 OE 0 (3 = 0. if q > 2*) such that H(x,u,v) 2 H(x,u,v)u+ H(x.u,v)v-3 for all (x,z); (Hs) there are a (1, 2)and 5 (0, 1/2)su«h that H(x.u,v) ?41u[ a -SAv 2 for all (x, z)" (u) if q 2-, th« U,.(z, ), ,[ -(v + ) fo 1 (z, ). With these assmnptions we have the following three results, for the cae when p < 2. Theorem 1.3. Suppose that (Ho) holds with p < 2 and q 2. If H(x,z) also satisfies (Ha) - (H), tben (E) has at least one nontrivial solution. Theorem 1.4. Suppose that H(x, z) is even in z and (Ho) holds with p < 2 and q 2. IfH(x,z) also satisfies (H4)-(H), then (E) has a sequence (z) ofsolutions with negative enewies I(zn) going fo 0 as n oe. Theorem 1.5. Let (Ho). with p,q (1, 2), and (Hs) be satisfied. Then (E) bas at least one nontrivial solution. If. in addition. H(x, z) is even in z, then (E) bas a sequence (zn) of solutions with negative energies I(zn) going to 0 as n oe. Finally, we consider the ce when p = 2, which presents some sort of resonance. Assmne (HT) there exist b0 _< 0 < a0 such that Ro(X, z) := H(x, z) - (aou + boy 2) = o(z] 2) z 0 uniformly in x; (Hs) there exist « (1, 2), aoe la0, Oe)«(-A), such that R(x,z) := u(, )- t« 10(, )1 (l+ll-+vV -) o n(, ) lv-(1 + The position of the numbers a0, aoe, b0 with respect to the spectrmn «(-A) plays a very essential role in the next result. For that marrer, let i,j, k be nomegative integerssuch that Ai =min{A «(-A) A > a0}, A =max{A «(-A) A < {j if a = a0, [= j+k-i+l if a > a0. Now we can state out last result. Theorem 1.6. Let (Ho) be satisfied with p = 2 and r < 1 + q/2. Assume tbat u(z,) i v« i ad atifi () d (gs). Th (E) ha at tat o« pai of nontrivial solutions if = 1, and infinitely many solutions if 2. The ces covered in Theorem 1.6 include some ymptotically linear systems. Such systems bave been studied in [5], [6] and Silva [13]. However, their results are hot comparable with the ones obtained here. 2976 D G DE FIGUEIREDO AND " . H. DING We organize the paper as follows. In order to establish multiplicity of solutions we need some new abstract propositions on critical point theoD- for strongly indefinite functionals, which will be provided in Section 2. These propositions are based on certain Galerkin approximations, and we emphasize that the functionals do not satisfv the usual Palais-Smale condition. In Section 3 we study svstems that are _-uperlinear in the variable u. and prove Theorems 1.1 and 1.2. In Section 4 we consider systems that are sublinear in the variable u. and prove Theorems 1.3. 1.4 and 1.5. In both Sections 3 and 4. the variable v can have subcritical grooEh as well as supercritical grooEh. Finally. in Section 5. we consider a special asymptotically linear svstem and prove existence of multiple solutions. "2. CRITICAL POINTS FOR STRONGLY INDEFINITE FUNCTIONALS Let E be a Banach space with norm [[-[[. Suppose that E has a direct sure decomposition E = E @ E 2 with both E and E 2 being infinite dimensional. Let P denote the projection from E onto E . Assume (e) (resp. (e)) is a basis for E (resp. E2). Set X := span{e, ..... .en} + E 2. ,\' := and let (X) ± denote the complement of X m in E. For a flmctioIml I Ci(E, 1) we set In := Il \,, the restriction of I on \-n- Recall that a sequence (z j) C K is saidtobea (PS) sequenceifzj Xn n --. _-'c, I(z) c and I' (--) Oas j :. If an)" (PS) e sequence bas a convergent subsequence, then we sav that I satisfies he (PS) condition. Denote the upper and lower level sets. respectively, by la = {z E " I(z) _> a}. I = {: E" I(z) <_ b} and I =/ ç I (denote similarly (I)«, (I) and /)). Wealsoset K = {z < E- I'(z) =0}. /Q = Kç/. K =KçI and /C =/Q c/C . Proposition 2.1. Let E be as above and let I C(E, IR) be even u, tth I(0) = O. In addition, suppose that. for each m Ç N. the conditions below hold: Ii) there is Rn > 0 such that I(z) < 0 for all - ,12) there are r, > 0 and a, -- such that I(z) > a, for all z Ç (X"-I) ± with Ilzl[ = rn" 113) I is bounded from above on bounded sers of X': I_ 0 if c >_ O. any (PS) sequen«e (:) has a subsequence along whi«h zn -- z Then the functional I has a sequence ( ck ) of critical values, with the property that Ck -. Remark 2.1. This proposition is more or legs known if the condition (I_) is replaced bv the (PSt* condition (cf. [1], [9]), or bv the usual Palais-Smale condition, that is. any sequen«e (:k) C E such that [I(zk)[ <_ c and I'(zk) -- 0 bas a convergent subsequ«nce (cf. [31). Proposition 2.2. Let E be as above and let I C(E.I) be even. A.sume that I(O) = 0 and that, for each m N. the two conditions below hold: 115) there are r > 0 and a. > 0 such that I(z) >_ ara for all : X m with ri6) there is b >0 u'*th bm -- O uch that I{:) <_bru for all : (X-) ±. STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2977 Moreover, suppose that either I satisfies the (PS) condition for ail c > O. or that the condition below holds: (17) inf I(/Ç) = 0. and. for allc >_ O. any (PS); sequ«nce (z,) bas a subsequence alon9 which zn z Çc witb z = 0 only if c = O. Then I bas a sequence (ck) of positive critical values satisfying Ck O. Proof. Let E be the family of symmetric, closed subsets of E{0}, and let E N U {0, } denote the Krasnoselski genus nlap. Set % := sup where rU := {A r- A c X a,,d (A) >. + ,,,}. Fix m N. The Borsuk-Ulam flmorem implies that A (xm-) ± 7 for eaçh A Z TM. It follows from (Ia) that inf I(z) < sup I(z) < b,. zA z( k'm- 1 ) / On the other hand, since and so, by (I5), we obtain Therefore, (2.1) çX, ) = n+m. one bas S,, " := OBr,f-)X,rî E E n , inf I(z) >_ ara. ara <_ c n _ A standard deformation argument, using a positive pseudo-gradient flow. yields the ?72 OO ?72 existence of a sequence (z)=, with z X satisfying 1 1 [I(z) - cl and , m ? 71 «an sun,e that I(z) cm as n . So, (z.,) is a (PS) sequen«e with (2.2) a c b. Now, if we assume that I satisfies the (PS)gcondition for c > 0, then the conclusion m follows. Next, suppose instead that (I7) holds. Then, along a subsequence, z z as n with I'(z) = 0 and 0 < I(zm) Cm. Finally, by (2.2), I()Sb o, and the proof is complete. Proposition 2.3. Let E be as above and let I C (E,N) be even wth I(O) = O. Suppose, in addition, that the three conditions below hold: (Is) the «e e N ., > 0 h tht I() (I9) there is b > 0 such that sup I(E 2) b: (Ira) any (PS), c > O. sequence (z) bas a subsequence along which z Çc and Pz Pz. Then I bas at least one pair of nontrivial critical points if = 1. and infinitely many ctical points if f > 1, with positive critical values. 2978 D. G. DE FIGUEIREDO AND Y. H. DING Pro@ Let E, % E T and c be as in the proof of Proposition 2.2. As before, by (I8) and (I9), we obtain " < b for all n N and tri = 1,.-- ,. and we find sequences z G X such that, going to subsequences if necessary, I(z) c and I,,(z n ) 0 as n , with b Cl C2 "'" Ct 2 a. Using (I0), we can assume flrthermore that n' z G « for m = 1,--- , g, as n . If g = 1 the proof is complete. Consider (> 1. Let F = {z G " I(z) > 0}. aregoingtoprove that F is an infinite set. Arguing by contradiction, we suppose that F is finite. Çhoose 0 < g < a b < u satisfying tt < inf I(F) sup I(F) < u. Let k G N be so large that 0 A := QF, where Q E X denotes the projection. Then A is also finite, and (A) = 1. By the continuity of % for all 5 > 0 small, (N(A)) = (A), where N(A) = {z X dist(z,A) 5}. Set c = (.4)(x) . Sn« (A) c c ,,d Q "C V(A). t onow rom th properties of that (C6) = (N(A)). remark that Q = p1 + (Qk _ p) and that the range of Q - p1 is k-dimensional. So bv virtue of (I10), we conclude that, for all c 0, any (PS) sequence (zn) has a subsequence along which zn z G « and Qkzn Qkz. Hence there are n0 G N and a > 0 such that for all n n0, []I,'(w)l ] a for all w G (I) C, where C = C6 X. Bv a standard deformation argument, we can then construct a sequence of odd homeomorphisms q X X such that ,, ((&) k C?) C (Z,) (cf. [12]). For n0 sufficientlv large, we can suppose that p<% % _'"_c n<u forallnn0. Let G G E be such that inf I(G) > ( + c)/2. One then h and (,»,(c\c')) = (c\c') >_ (c) -(c') > + t. - ")'(Ce) > r, + t. - . Thus rl,(G\C') Z -1 and v <_ inf I(rl,(G\C') ) <_ %t-,. One finallv, con,es to t--1 r, _< % < r,, which is a contradiction. [] From now on we turn to the system (E). We denote by I" It the usual Lt(Q) norm for ail t [1, oc]. For q > 1 let Vq = H(Q) if q <_ 2* and Vq = H(Q) çlLq(Q), the Banach space equipped with the norin IIvllvç -- (IVvl ÷ IriS)1/2, if q > 2*. Let Eq be the product space H(t2) x I with elelnents denoted by z = (u, v). We denote the norm in Eq by Ilzll« = (Iwl ÷ Ilvll) 1/2. has the direct sure decomposition Eq = E @ E +, z = z- + z + where E-={0}xVq and E +=H(t2) x{0}. STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2979 For convenience, we will write z + = u and z- = v. t/eca.ll that by (A,),EI we denote the sequence of eigenvalues of (-/N,H{II(H)). Let e,, le,12 = 1. be the dgenflmction corresponding to A, for each n E N. Clearly, e + := (e,, 0), r E N, is a basis for E +, and e = (0, e,), n N, is a basis for E-. Suppose that the assmnption (Ho) holds. Then (2.3) So the functional (2.4) (2.5) and H(x,z) <_ c(1 + lu[ 2. + I¢,1 ) o n («,z). il £ I(z) := ([Vul 2 -IVvl 2) - H(x,z) (2.6) ] H,,(x,z)b ./o H,(x,z)b for ail By the Sobolev elnbedding theoreln and using interpolation, we obtain that u u in L t for t [1, 2*) and v v in L t for t [1, q). Noting that ]H(x,u.v) %(1 + u -I + ]v -) with 2.(r- 1) < q, (2.5) follows easilv since u in L ,v vinL 2.(-1) andç H() C L 2.. Next weseethat (2.6) is clely true when ¢ L . In general, for a ¢ t we proceed as follows. Let ç e L oe with ¢ in L q as m oe. So IZ(H(x,z) - H,(x,z))¢I = I£(H,(x,z) - H,(x,z))( + (¢- m)) , and using (Ho) we see that this expression is less than the following sure: + , (> - ml + >vlm -', + va-m - which by its turn is estimated bv since (z) is bounded in Eq and L is dense in Lq. So (2.6) is proved, and it follows that I'(z)w + I'(z)w for ail w Eq is well defined in Eq. Moreover, I ¢ Çl(E'q,]l), and the critical points of I are the solutions of (E). Lemma 2.1. If (Ho) holds, then I' is weakly seqttentially cotinuous, hat is, I' ( z, ) -- l'(z) provided z, -- z. Pro@ If q < 2* this statenlent is well known. Assume now hat q > 2". Let z, -- z in Ev Çlearly, for all w = (ç, ) Ev we bave So it renmins to show that [ H=(x,z,)o -- [ H=(.r,z)o for all qO Ha(t ) J 2980 D.G. DE FIGUEIREDO AND Y. H. DING 3. THE CASE p > 2 Throughout this section let (Ho) be satisfied with p > 2, and assume that and (H2) hold. Observe that, by (H2), there exists R > () such that H(z,z) > 0 whelever Izl > R. This, jointly with (H1), implies (3.1) H(x,z) > «(lui" ÷M)-c2 foran (see [10]). This, together with (2.3) and (H2), shows that (3.2) u <_ q and /3 _< q. Moleover, by virtue of (3.1) and (H..,), we lnay assulne, without loss of generality. that (since p > 2) (3.3) « > 2. 1 e 2 e + for ail n (5 N. So Now we set. E 1 = E-, E 2 = E + and e n = , e n = Eq = E1 ( E2. Consider the flmctiolml defined by (2.4), which has the properties stated iii Section 2. Lemma 3.1. ,4ny (PS) sequence is bounded. Pro@ Let z, G X, be such that I(z,) c and I'(z,) O. -- n), we Cae 1: q < 2. h thi ca E = (n()) . m" (n), for ' := ( , have (3.4) =(_71 _ 1, )lVul ÷ ( 1 _ __ )lVvl ÷ (;H"(x'zn)un + 1--Hr(x'zn)vn - H(x'zn)) -cloe 1 £ IwI 1 1)lwl-c» ->(- ) +(;- If q < 2. then (3.2) shows that pe < 2, and so I1=11 -< e(a ÷ I1=11), which implies that (z,) is bounded in Eq. Assulne q = 2. hvokillg (3.2), we get u 2. and so I c(1 + [lz[lq) by (3.4). Since ti(x,z) > 0 for all [z] large, and one sees that [z[[ N c(1 + [[zllq). Hence, (z) is bounded. Case 2: q > 2. Note that in this case = p > 2 in (H). So (3.5) I(z,)- -I'(z,)z = (7_H(a',z)z- H(x,:)) # - 1) f H(x, z) - _>( C, which, together with (H2), yields STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2981 Using (Ho), we get (3.7) Next we estimat.e the integrals in the right, side of (3.7). Since 2.(1»- 1) p, we have that 0 := /(1 + a - p) 2*. Using tho H61dor inequality, the Sobolev embedding theorem and (3.6), we obtain Similarly, since z - 1 < ffC., we bave 1 < w := /(1 + [3 - z) < 2*, and hence Il-ll[ I' - Therefore, usiug th est.ima.l« in (3.7), wc oh/aih _ , l+(p-)/ Iç,t < «(1 + It + IIzll+<-')/n) (3.s) Since IVv,] = -l'(z,)(O, v,) - fa H-_(a', z,)z, + fa H,,(x,z,)u,, and using (3.5) and the above argmneuts, we obtain (3.9) IV,l Recall that, in view of out sumptions, (p- 1)/ 1/2., (r- 1)/ < 1/2.. and fl = q if q > 2*. Hence, it follows from (3.6) and (3.8)-(3.9) that (z) is bounded in Eq. Lemma 3.2. Let z, X,be a (PS) sequence. If q 2", then (z) contains a convergent subsequence. If q > 2", then te is a z Eq such tat. along a subsequence, z z and l'(z) = 0 and l(z) c. of. By Lemma 3.1, (z,) is bounded. can assmne that z, z in Eq, z z in (L()) 2 for ail 1 s < 2*, and z(x z(.r) a.e. on . It follows from the weak sequential continuity of I' (see Lemma 2.1) that l'(z) = 0. Since I'(z) 0, we obtain ). Using (H0) and the H61der inequa.lity, we obtain the estimate where is as in the proof of Lemma 3.1. Hence I1 Iul, -hih implies 2982 D. G. DE FIGUEIREDO AND Y. H. DING in Eq for ail z E Eq. Moreover, using again (Ho) and the H61der inequality, we estimate _< c([ç -- Pnçll + [n[ç-ll ç - Pç[ç+ ]ç[ç-l[ t' - Pat'[q) + 0. On the other hand, I[(z)(O,v - Pv) + [. H,,(.,z)(v- Pv) VV)L + Lebesgue's theorem and the weak sequential continuity of H:(«,-) (see the proof of Lemma 2.1) yieht ] Vv, - liln sup ] Vv ] = linl inf ( £ H: (,r , z ) zn - £ He ( x , z ) Z )noe noE 0, i.e., IVvl 2 lira sut,,, V*'I. This, together with the weak lower semicontinuity of norms, implies Vvu + [Vvlu. So v + v in H(). Therefore, if q 2*, we obtain that, along a subsequence, z + z in Eq and consequently I(z) = c. Next a.ssume that q > 2*. Observe that hence, Lebesgue's theorem then yields I(z) -c= liif H(x,z,O - £ H(x,z ) >_0, that is, I(z) >_ c. Lemma 3.3. If(H3) also holds, there are r,p > 0 such that inf I(OBrE +) > p. Pro@ By (Ho) and (H3), for anv e > 0. there is c« > 0 such that H(.. . 0)< el,[ 2 + c«M 2". Hence, 1 and the conclusion follows easily. Let e E E + with IVel = 1, and set Q= {(se, v) 0 _< _< q, Ilvll <_ r2I. STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2983 Lemma 3.4. If(H3) also holds, there are rl,r2 > O, wzth rl > r, such that I(z) <_ 0 for all z Ç OQ. Pro@ By (H3), I(z) <_ 0 for all z (5 E-. By (H2), I((se, v))< 2 [7v]-c1 (lel÷lvl)÷c2 The conclusion follows since > 2. We are now in a position to prove Theorem 1.1. Proof of Theorem 1.1. Lemmas 3.3 and 3.4 soEv that I has the linking geometry. Let Qn := Q 0 Xn, and define ch := inf maxI(3,(Qn)), where Fn := {? e C(Q,X,) : /IOQ, =id}. Then p_< c, _< := supI(Q). A standard deformation argument shows that thcre is z E X such that II(z)-cl < lin and I{I.(Zn)[I _< l/n. So we obtain a (PS): sequence (Zn) with c e [p, c]. Lemma 3.2 implies z -- z with I'(z) = 0 and I(z) >_ c. The proof is complete. [] We now consider the multiplicity of sohltions using Proposition 2.1. Lemma 3.5. I satisfies (Il)- Pro@ Using (H2), we obtain I(z) - z < lVul - 1]Vv] -c (lui a + Iv] ) + C 2 . Since all norms in span{e,.-- , e,,} are equivalent, we obtain I(z) <_ -(«31Vu] -2 )lVu]- (-]Vv]-- «l]V]) --C 2, for ail z = (u,v) G X TM -- span{el,-.-,e} x 1. So (It) follows easily. Lemma 3.6. I satisfies (I2). Pro@ Since (X) ± C H(ft) and H(ft) embeds compactly in LP(ft), we have that r > 0 and r 0 as m o, where (3.10) rh := sup ; see Lemma 3.8 in [14]. For z = (u, 0) (X') ±, it follows fi'om (Ho) that 1 iÇtl2 2 _ ci itl p _ C I(z) = 1 IVl- H(x,,O) _> - 1 -2 Setting r, = (pcr) 1/(2-») and a, = (p - 2)r/2p - c2, we corne to the desired conclusion. [] Proof of Theorem 1.2. Since H(x, z) is even in z, I is even. Lemma 3.2 shows that I satisfies the assumption (I4) of Proposition 2.1. Lemmas 3.5 and 3.6 show that (I) and (I2) hold. Clearly (I3) is also true. Therefore by Proposition 2.1, there is a sequence (z) C Eqsatisfying I'(z) = 0 and I(z,) -- o. The proof is complete. [] 2984 D. G. DE FIGUEII:IEO AND Y. H. DING 4. THE CASE l» < '2 Throughout this section we assume that (Ho) is satisfied with p (1. 2). also suppose that (Ha) - (H) hold. Let Eq = E 1 E 2 be as in Sectim 3. Consider the flmctional Lelllllla 4.1. Auy (PS) scqu.«m'e (Zn) ha.s a subsequence collverging wea£1y to a criti«al poi'nt z qf J with .I(z) c. a,d z = 0 only if z= (1 i Eq. Pv@ The l)r()of is divi(h,d ino wo parts. Pat 1. The sequ(,n('c (z.) is bom.led in Eq. By (Ha) it follows that Hence IVu,,l c(1 + IIzllq). If v > 2, we also got IV,l c(1 + llzllq). If v = 2, wc uso (fla)and tire fa«t that Içl OE a,ll i, or,ie to obtain Thus, if q N 2*, thon (z.) is bounded in Eq. Assmne next that q > 2*. It follows ri'oto (Hi) t hat in Eq also in the case when q > 2*. Part II. %% can now suppose that z. z in Eq, On Z in (L(ll)) for all 1 N s < 2*, and z(m z(m a.e. in m Q. It follows that z is a critical point of J. As in the proof of Lonnna 3.2. using (H0) and Using (Ho), we have /-/,, (.r. + fa Ho(x, z.(v - P. STRONGL INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2985 Colsequent ly, (4.2) ('v, (v-- v,))L, = i Hr(.'r, Zn)( -- v) + o(1 ). Thus if q < 2", it follows from (4.2) that {Vt,[ and so zn z. This proves lhat .l satisfies he (PS);condition in this case. and that J(z) = c. Consider next q k 2*. The weak sequential contilmity of H,,(.r,-) (see the proof of Lemma 2.1) yields fn H(.r, zn)v j; H,(.r,z)v. By (H), fn(z):= H,(.r, zn)t' + 76(1v1+11 ) 0. Using the fact that [v] ]v[ and [t,] 1[, and applying Fatou's lemma to the sequence (f), we get Using this estilnate in (4.2), we obtain that IV,] lilnsUp,,oe V'I, whi«h implies that. t, ' in H). In Ol'dm to Colwlud« that .lz) c, we use the est ilnat.e .l(z)-.J(z) = (H.r, zn)- Ha',z)) + o(1), and so z 0. Remark 4.1. In a similar way, using even silnpler argulnents. OlW checks that. if (H0) holds with p, q (1, 2), .1 satisfies the (PS) condition for all c. Remark 4.2. Let ,], = J x- denote the restriction of J on X . As in Lemlna 4.1, it is hot difficult to check that. if the sequence (z) Ç Eq, with z G X , satisfies J(z) c and (z) 0 as m , then it possesses a subsequence converging weaklv to a critical point z of J with J(z) c. and z also bave, as in elnark 4.1, that. if (H0) holds with p. q G (1, 2), then anv such sequence bas a convergent subsequence. Lemma 4.2. There is an R > 0 such that J(z) 0 for all z = (v,O) with lzll ool. y (,,), w hv (,,..0) «(1 + ). and the lelllllla follows, SillCe p < 2. Lemma 4.3. Fore > 0 small the is p > 0 such that J((ee.v)) p for all v . whe el is the eigenfunction correspodin9 to the first eigem, alue of (-. (t)). Pro@ Bv (H.5), for e > 0 small. H(z, ee,v) k 24eeî- akv2: hence, , 1 'V'" - > ('[e" - '-)" J((ee.v)) = H(.r, ee. v) + _ The conclusion follows. 2986 D.G. DE FIGUEIIREDO AND Y. H. DING are llOW ready to prove Theoreln 1.3. Proof of Theorem 1.3. Recall that X TM span{el,--- , e,} x l,q, and consider the restrictions J, as defilmd iii Rema.rk 4.2. Set Dn = Bn ç E 2 = Bn fq (HH(12) x {0}) and Dru = DR fq X m, where R > 0 cornes from Lemma 4.2. Define ç,n := inf max J(7(D,n)), where Fro := {7 C(D,, S') : 7(z) = z forall z OD,,}. It is well known tiret 7(Dru) II" ¢ ¢ for all 7 Fro, where II" = {(«o.0)} x 1 with « > 0 slnall. Invoking Lemma 4.3, we fix an ê > Il so snmll that there is p > 0 satisfying inf./(ll') p. Then we have t' c. b := max J(Dn). The well-known saddle 1,oint lheorem (cf. [12] or [4], [14]) implies that there is z, G X m satisfying ]J(zm) - c] l/m and ]].[(zm)]] 1lin. Now by virtue of Remark 4.2, along a sul,sequence, z z with ,l'(z) = 0 and z ¢ 0, ending the proof. "Ve llOW turn to the proof of Tlworelns 1.4 alld 1.5. Lemma 4.4. If. in addition. ")'3 = 0 in (Ha), then J satzsfies (I5)- Proof. It follows from (Hs) that J(z) > c I I + ( (4.3) Since a < 2, the result follows in the case when q 2*. Next consider q > 2*. Suppose (I) does hot hold. Then for any r > 0 there is a sequence zj X TM such that IIzll = al,d J(z) O. It fonows frein (.a) with z = z, and for r slnall. that Içl= 0 and [çvj[= 0. Ail this ilnplies that sumption (H0) and t he fact t hat (ai) lies in a finite-dilnensiolml subspace, it follows that fH(x, zj)uj O. Colasequently, by (H4)with 73 = O, f Hv(x, zj)vj O. This, jointly with (Ho), yields 0. Hence, zj 0 in Eq, which is a contradiction. Lemma 4.5. J satisfies (16) Proof. By (Ho), H(a:.,,,0) <_ c(lu[ + I**l), alld so, for u (X'-I) ±, Olle hg8 (Cll']p- [ l[v[)lVu[ = + (Cl _ 1 where Vin was defined by (3.10). Let b := Then 0 < b 0 and J((u,0)) b for ail (a,0) STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2987 Proof of Theorcm 1.4. Since H(x,z) is even in z, d is even. If q < 2", then d satisfies the (PS) condition for all c (see the proof of Lemma 4.1). If q > 2*, then, using assumption (H4) applied to a critical point z, we obtain 3(z) = 3(z)- £(z)(**,-v) > 1 1 Iwl + 0. _'2 ,t - This, jointly with Lennna 4.1, shows that (If) is satisfied. It follows from Lennnas 4.4 and 4.5 that J satisfies (I5) and (I). Therefore, the desired conclusion follows. Finally, we prove Theorem 1.5. Proof of Theorem 1.5. The proof of the existence of one nontrivial sohltion is sim- ilar to that of Theorenl 1.3, using Ptemark 4.2 and LellllllaS 4.2 and 4.3. The other conclusion tan be obtained along the lines of the proof of Theorell 1.4, using Remark 4.1 and Lennlms 4.4 and 4.5. [] 5. THE CASE p = 2 hl this section we alwa,vs assume that (Ho) holds with p = 2 and r < 1 + q/2. We also suppose that (H7) and (H8) are satisfied. We will apply Proposition 2 3 in order to prove Theorem 1.6. Thus, set /72 =span{eï, --- ,e} : /I ----spall{el,'--,et-} X Ç, /71 - /q(E '2, and X £ /71 (t)span{ï,-- + (7,"" (;} ,ek, , 2 + forl<< 1 + for n E 1, and e n= en+i_ 1 One may arrange the bases as % = %+ _ _ 2 + for g < n < f+i- 1, and 2 - 2 - forg-j<n<t e . %=%_t %=e,_ g--j, e n = en_t+ j -- -- for rt > f + i- 1. Consider the filnctional I given by (2.4). Lemma 5.1. I satisfies (Is); that is, there exist r, a > 0 such that I(z) >_ a for all z e X t with Ilzll = - Pro@ Let z = (u,v) X e. Since v Ç span{el,.-- ,ej}, we have v Ç L . By (Ho) and (H7), for any e > 0, there exists ce > 0 such that /0(, ) < ell 2 + c,(M 2" + Iris). Thus 1 I(z) = (IVul - a, olul ) - - ([x7vl - b01vl) - J Ro(x, Z) '(_ ) + ( ),-«- (: + > _ ao V 2 1 -bo -2 Now the conclusion follows easily. Lemma 5.2. I satisfies (I9); that is. sup I(E 2) < oc. 2988 D.C. DE FIGUEIIIEDO AND Y H DING Pro@ For z E E 2 we ha'«', using (Hs), that IV'loe- f/(', z) [() = - '2 which ilnplies that l(z) 0 for ail z e E 2 with IIz[[«large. Lenlma 5.3. Let c > O. Thcn any (PS)c sequewe ts bounded. Pv@ Wc decompose H] (t) H(t)=U-6 +, t,=,-+u +, «hcre - = span{e,---,eh} and U + is the orthogonal complelnent of U- in ,l(t). Let (z) bc a (I'S)g sequence. Using the expression of I,': +2 plus (Hs) and the Hi;lder inequality, we obtail a--1 + A+ - where r = q/(l+q-ç). By sumption. 1 < r < 2. It then follows Kom the Sobolev embedding theorems that + - Similarly, we deduce that The two previous inequalities ilnply the estinmte Using the expression of H given in (Ha), and recalling that I(z) > 0 for large we obtain (5.z) } .- -_ Next using (5.2), assulnption (Hs) and (5.1), we obtain The combination of (5.1) and (5.3) implies + Since a < 2 and 2(r - 1) < q. we see that (z,,) is bounded. Lemma 5.4. I satisfies (Io). STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2989 Pro@ Let (zn) be a (PS); sequeuce wiîh c > 0. Using Lelmlm 5.3, an argulient similar to that of Lçlnlna 3.2 shows that along a subsequence zn --" z E/Cc, we bave ,, -- z in Hot(tl). Since E1 C H01([), we have Pzn Ptz. [] Pwof of Theorem 1.6. Since H(x, z) is evell in z, I is even. Bv assmnption. I(0) = 0. Lemnlas 5.1, 5.2 and 5.4 show that I satisfies (Is) - (It0)- Now Proposition 2.3 applies, aiM the proof is COlnplete. [] A('KNOWLEDGMENTS De Figueiredo was suppolted by CNPq-FAPESP-PIR()NEX. Ding was supported by the Special Fuuds fol lkla.iOl" State Basic Research Projects of China. the flmds of ('AS/('hilm 119902. lfl01Nl{-I, and the CNPq of Brazil. EFERENCES 1. T. Bartsch, lnfinitely many solutions of a symmetric Dirichlet problem. Nonlinem- Anal. TMA, 20 (1993), 1205 1216. MR 94g:35093 2. T. Ba.rtsch and I). G. De Figueiredo, Infinitely mang solutions of nonlinear elliptic systems, Progress in Nonlin«ar Differential Equations and Their Applications, vol. 35, Birkh/iuser, Basel/Switzerland. 19.q9. pp. 51 67. MR 2000j:35072 3. V. Benci and P. Rabinowitz. Critical point theorems for mdefinite functonals, lnvent. Math. 52 (1979), 241 273. M1R 80i:58019 4. K. C. Chang, lnfinite-Dimensional Morse Theory and Multiple Solution Problems, Birkh/iuser, Boston, 1993. MR 94e:5023 5. D. G. Costa and C. A. Maga.lhfies, A variational approach to noncooperative elliptic systems, Nonlinear Anal. TMA 25 (1995), 699 ïl5. hlR 96g:35070 6. D. G. Costa and C. A. Magalh£es. A unified approach fo a class of strongly indefinite func- tionals, .J. Differential Equations 125 (1996), 521-547. gin 96m:5N161 7. D. G. De Figueiredo and C. A. hla.galh£es, On nonquadratic Hamiltonian elliptic systems, Adv. Differential Equations 1 (1996), 881-898. Mil 97f:35049 8. D. G. De Figueiredo and P. L. Felmer. On superquadratic elliptic systems, Trans. Amer. Math. Soc. 343 (1994), 97-116. Mil 94g:35072 9. Y. H. Ding, Infinitely many entire solutions of an elliptic system with symmetry, Topologica.1 Methods in Nonlinear Anal. 9 1997), 313-323. hlR 99a:35062 10. P. L. Felrner, Periodic solutions of "superquadratic" Hamiltonian systems, J. Differential Equations 102 (1993), 188-207. hlR 94c:58160 ll. J. Hulshof and Pt. van der Vorst, Differential systems with strongly indefinite variational structure, J. Funct. Anal. 114 (1993}, 32-58. hIR 94g:35073 12. P. H. Rabinowitz, hhnimax Methods in Critical Point Theory with Applications to Differential Equations, C. B. M. S. vol. 65, Amer. Math. Soc., Providence, RI. 1986. Mil 87j:58024 13. E. A. B. Si|va, Nontrivial solutions for noncooperative elliptic systems af resonance, Electronic J. Differential Equations 6 (2001}. 267-283. Mil 2001j:3509ï 14. M. Wiilem. Minimax Theorems, Progress in Nonlinea.r Differential Equations and their Ap- plications, vol. 24. Birkh/i,ser. Boston, 1996. hlR 97h:5037 IMECC-UNICAMP. CAIXA POSTAL 6065, 13083-970 CAMPINAS S.P. BRAZIL E-mail address: dj airo@ime, unicamp, br INSTITUTE OF ,MkTHEMATICS, AMSS. CHINESE ACADEM5 OF SCIENCES. 100080 BE1JING. PEO- PLE'S I:{EPUBLIC OF CHINA E-mail address: dingyh¢math03, math. ac. Ch TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 355, Number 7, Pages 2991-3008 S 0002-9947(03)03279-3 Article electronically published on Match 17, 2003 STABILITY OF SMALL AMPLITUDE BOUNDARY LAYERS FOR MIXED HYPERBOLIC-PARABOLIC SYSTEMS F. ROUSSET ABSTRACT. We consider an initial boundary value problem for a symmetriz- able mixed hyperbolic-parabolic system of conservation iaws with a smail vis- cosity e, u[ + F(u«)x = e( B(u «)uex)x. When the boundary is noncharacteristic for both the viscous and the inviscid system, and the boundary condition dissi- pative, we show that u « converges to a solution of t he inviscid system before the formation of shocks if the amplitude of the boundary iayer is suflïciently smaii. This generalizes previous results obtailmd for B invertible and the lincar study of Serre and Zumbrun obtained for a pure Dirichlet's boundary condition. 1. INTRODUCTION We consider a one-dimensional systenl of conservation laws with a small param- eter « set in the dornain x > 0. (1) u t + F(uS): = e(B(uS)v)x, x > O. t > O. where u e G n and F H n, B H nxn. ç will assume that F and B are smooth (Cm). We add to this system an initial condition us(0, x) = u0(x) and a boundary condition that we will detail later. sume that the eigenvalues of B have nonnegative real part and that the tank of B does not depend on u. Ve will denote it by r, 1 r n. Note that B is not necessarily invertible. We are interested in the limit of u e when « tends to zero. çX expect that u e tends to a solution of the inviscid problem: (2) ., + F() = 0 with sonle bomldary conditions to be determined. At first we make the natural sunlptions to ensure the well-posedness of the Cauchy problem for (1) [6]. There exists a change of variable u v(u) with inverse u = 9(v) in which the systenl can be rewritten as (3) g(v)t + f(v) = e(b(v)Vx) with the following properties: (H1) b(v) is block diagonal, b(v) = 0 bi(v) " with b (v) e GL(). Received by the editors January 30, 2002 and, in revised form, December 13, 2002. 2000 Matheraatics Subject Classification. 35K50, 35L50, 35L65, 76H20. ()2003 American Mathematical Society 2992 F. ROUSSET (H2) dg(v) is lower block diagonal. dg(v)= ( v O) (,) " with t)(v) E GL,-(). Bv analogy to the terminology in gas dvnamics, we shall refer to t' as the primitive variable. Next we assmne that (1) is symmetrizable mixed hypert»olic-parabolic: (H3) there exists a positive definite symmetric (u) such that (1) E(u)dF(u) is symmetric, (oe) (,)(.)x. x l()Xl =, x , where > 0. mM - stmMs for the scalar product of . k2, denote by v = (w, z) the corresponding block deeomposition of ç. Note that since [(v) =dF(u)dg(v), b( v)= B(u)dg(), setting S()=dg(v)t(v), we get that (H3) is eqnivalent to (H3') There exists S(v) such thm, (1) S(ç)dg(v) is positive definite svmmetric, (2) S(v)df(v) is svmmetric, (3) S(,)b(,)x. x > .ll , vx = e $$% point out that (tt3)(1) implies that the inviscid svstem (2) is hyperbolic. 3Io» over (H3)(2) in,plies that dgtd9 is i,h,ck diagonal (see [11], Lemma. 4.1). Hence thanks to (H2), ce get S(v) where S(v) is positive definite symlnetric. Consequently, writing the block de- composition of df as we get flore (H3')(2) that Sw(v)h(v) is symmetric. This means that the system obtained from (3) by removing the second equation is symmetric-hyperbolic. Finallv. we also assume that the hyperbolic and parabolic modes do couple: (H4) The kernel of B does hot contain anv eigenvector of dF. The structural hypotheses (H1-H4) are verified by many physical equations as those of compressible gas dvnamics and nmgnetohydrodynamics. Next we make hypotheses to deal with the initial boundary vahle problem. We focus on the case of a noncharacteristic boundarv. We assmne that the boundarv is noncharacteristic for both the viscous (1) and the inviscid (2) systems: (H5) dF(u) and ](ç) are nollsingular. Note that an inflow or outflow boundary condition nmkes the boundary noncharac- teristic in most cases for the Euler and Navier-Stokes equations. These boundarv conditions have a physical meaning since they appear iii problelns with aperture. such as in oil recoverv. The analysis of an impermeable boundary xvould be different since in this case the boundary is characteristic. We denote by q the number of eigenvalues of positive rem part of dF(U), and by p the nmnber of eigenvalues of positive real part of h(v). An initial boundary STABILITY OF BOUNDARY LAh ERS 2993 value problem for (1) nee(ls p + r scalar indepen(lent bomdary conditions, and an ifitial boundary lue problem for (2) needs q independent scalar boundary conditions. We deal with bounda.ry conditions fl»r (3) that are linear with respect to the primitive variable v. çV write the bOulidary condition for (3) as (5) Lv(t'O)=(l'") (''O)=9.z where l is a linear mal» that has rank p and 9 is a given constant. In the following, in order to make energy estinlates, we assmne that the boundary condition (5) is "dissipative": *(H6) fl > 0. Vv ç, VX = .¥ . such that Lv = 9- LA = 0. and wc have s(v)(v)x . x -lXl . There are physical boundary conditions in the form (5) that satisfv (ti6). The case of the isentropic gas dvnamics will bc stud;ed Iwh»w. Note that thanks to hyl,otheses (H I-Ha), we have p ç q p + r ([12]. Corollary 1). Hence in the case q < p + r. there is a h»ss of bOmldarv condition when tends to zero. It is due to a fast change of u in a vicinitv of the boundarv: the b«nmdarv laver. In the noncharacteristic cas« the size of tlw bomdary laver is ¢. When ¢ tends to zero. the expected t,ehaviour of ,.« is ([2], [1], [12]) ,.it(t. z) + U(t..r where u it is a soluti(,n of (2) with the initial condition ,t(0..) = u0(z) and some boundary conditions that we have to deternfine. U(t, z) is a boundary layer; it is a solution of a differential problem where the time is onlv a parameter: (6) (U)U' = (U + ,,'(t. 0))- F("(t.O)). U(t, +OE) = o. L,(U(t.O) + ,,"'(t. 0)) = g. Note that when r < , we have an algebraic differential system. This problem has solutions if and onlv if uit(t. 0) belongs to the subset C. where c = u+ ", u, u(+oc) = o. L,(U(O)+r+)=g. This sct C is called the set of residual boundary conditions. It was studied in the case r = n in [2], [4] and ill the gelleral Ça,S(? iii [11]. Assuming that u0(0) satisfies the boundary condition (5), i.e., (7) L(o(0)) = 0. we have 0(0) C (the associated profile ofthe boundary layer is U = 0). Moreover, thanks to (H1-H6) we can use [11], Lemma 4.2 and Theorem 1.1. C is a smooth submanifold in the vicinity of u0(0) that bas dimension q and that is transverse to the unstable subspace of dF(u0(0)). Consequentl3; thanks to a theorem of [8], there exist.s a continuous solution of (2) with the boundary condition uit(t, O) C defined on [0, T] for some positive small time T. Assunfing some higher-order compatibilities between "u0(0) and C, we can even get a smooth solution u. Usiug 2994 F 1ROUSSET the smne method as in [4], we can show the existence of an approximate solution of (1) in the fornl (8) such that where ^1 i=1 Lv(u«vP(t.O)) = O. uaPv(O.x) = uo(C). 0, + OF(, ) - eOx(B()Ox) = R ]]R¢E]IL[O,TI,L 2 __ Cg M. Our aire is to show that the truc solution a f is close to the approximate solution if the boundary laver is sufficiently weak. More precisely, Theorem 1 (Nonlinear stability). Assumin9 (H1-H6) and that uo 6 Hr (+ ), there eists > 0 such that if (9) sup (lO:U(t,')l + zlO:U(t,z)ldz ]O=U(t,z)ldz a, tG[0.T] then tf e -- ri int --* 0 (o) with () ((()«f(v)) ÷ S()b(@X. X _ OlXl , VX IR ' k(v)dg(v) skew-synmmtric. when ¢ -- 0 in L([O,T],L2). To prove this theorem, we actually need to start from a very accurate approx- imate solution -u app. Indeed, we will take M = 3 in the expansion (8). The con- struction of such a high-order expansion requires a lot of regularity on u i't (sec [4]). This is why we have to assume so much regularity on u0. actually get a more precise estinmte: I1(. « -- .«)(t, ")11= + «110=( « -- )(t, ")11 + «11( "« -- )(t, .) Il w,. C«- Our method can also provide estinmtes in L([0, T], H ) for anv s. The proof of Theorem 1 relies on energy estimates. We use the primitive variable v; hence we work on the form (3) of the equation. X combine the energy estimate of the totally parabolic case (r = n) [2], [4] with an energy estimate of Kawhima's type [6] and a careful study of the boundary values. A key argument of the proof is the following lemma of [la]: Lemma 2 (S-K [13]). Ass.uming (H1-H4), there eists a skew-sBmmetric Iç(.u) and a positive constant 0 such that ((Iç(u)dF(u)) Æ + (u)B(u))X. X OlXl =, vx , v.. u '(KdF+(K«F) t) where (KdF) Æ = . Note t, hat setting k(v) = dg(v) tK(9(v)) we ean rewrite this result STABILITY OF BOUNDARY LAYERS 2995 Iu [9], Lenmm 2 aud the estimates of [6] combiued with pointwise Green's lune- tions bouuds were Mready used to prove the nonlinear asymptotic stability of weak time-iudependent viscous shock profiles for (1). The asylnptotic stability of a time- indepeude,t profile of the boundary laver together with the stability of other non- line waves was studied in [10] for the isentropic gas dvnamics rewritteu as a p system in Lagrangiau coordiuates. Let us give an example of an application of our theorem. C, ousider the isentropic gas dvuamics where v : (p, v), p being he mass deusity and v the fluid velocity, pv ' pv + p(p) ' o (p) " Here we assume that u > 0 and that p' > 0 (hyperbolicity). The sound speed is c(p) = . (H1-H4) are verified: moreover, the eigenvalues of dF are v c and the eigenvalue of (the 1 x 1 matrix) h is v. Let us first consider an outflow bouudarv condition (12) (13) (14) where (t. 0) = v- with v- < 0. In this ca, l = 0, mM (H6) I)ccomes v-}Xl 2 <-lXI2; hence, it is satisfied. The compatibility condition (7) becomes vo(0) = v_. It suffices to impose Vo((I) + c(po(0)) 0 to get (H5). If we consider an outflow boundary condition v(t,0)=v_, p(t,0)= where v_ > 0, we bave 1 = Id, b:er i = {0}, and hence (H6) is true. The com- patibility condition bccomes po(0) = p-, vo(0) = v_ and hence we get (HT) if v_ -c(p_) O. Moreover, in the case v- -c(p_) > 0, we have q = p+r = 2; hence there is no boundary layer and the hypothesis (9) is always satisfied. For a more general discussion of the various boundary conditions for the non- isentropic gas dynamics, we refer to [12]. As in the totally parabolic case r = n, the smalluess assumption (9) in Theorem 1 is linked with the stability of the bomMary laver. In [12] an example of a large unstable boundary la.ver is given. To understand the mechanism of instability in the boundarv laver, we set 0 = . , z = , we fix some time T in app and we linearize about the leading terre of u app with respect to . We get the linear svstem Oou = Ldv(u(r, O) + U(T, 0))v(0, 0) = 0, (0, z) = o(z) £-u = (b(ut(T,O) + U(T,Z))U' + dB(ut(T,O) + U(T,Z))uU'(T,Z) - d((,0) + U(-,z)) . Here ' stands for 7" We will say that the profile of the boundary layer u it (v, O) + U('r, z) for some fixed "r is linearly stable if t.he solutions of this system tend to zero 2996 F ROUSSET when t tends to +o« The linem" stability is linked with the spectral stability as was show,, in [15], [9]. [141. Let us define the domain of/2,- as /9(£-)= u=dg(l/(T,z))v, v= . w e HI(]+), z e H2(]+), Lv(t.0)=0 z where V(r, z)is defined by g(V(T,Z))=iWt(T.O)+ U(r, z). In [12], it is shown that the essential spectrum of /2- is confined in {7.eA < (I} U {0} thanks t.o (H1-H5). In the unstable half-plane {7.eA _> 0}\{0} the spec- trum only consists of eigenvalues. Conseqn('ntly, a necessary condition for the linear stabilitv of the boundary laycr is that the operator/2- does hot have eigenvalues in the unstable half-plalm {7.e A >_ 0}(spectral stability). An Evmls flmction machin- ery was developed in [12] to find sufficient conditions of instability. In the first part, we show that spectral st.ai)ility holds for weak boundarv lavers. Theoreln 3 (Spectral stability). There exists 5 > 0 such that, assuming (HI-H6) (15) thon F_«. docs hot bave ei9cvalues i the unstable hall-plane {R.c A _> 0}. The proof also relies on energy estimates. We first give a direct proof of Theorem 3 because it seems more enlightening to present the main ingredients of the proof of Theorem 1 in the simpler linear time-independent setting of Theorem 3. This result is not used in the proof of Theorem 1. The result of Theorem 3 could be deduced from direct energy estimates on the rime evolutionarv svtenl (12), (13), (14). Nevertheless it is interesting to study the spectral stability since ve can expect that, as in the totally parabolic case, the sharp assmnption of spectral stability implies the nonlinear convergence result [5]. Note that out result of Theoreln 3 (obtained bv a different method) implies the result of the appendix of [12] where only Dirichlet's boundary conditions were considered for (1). In the second part, we give the proof of the full nonlinear stabilitv result of Theoreln 1. 2. SPECTRAL STABILITY In this section, we prove Theorem 3. We studv the eigenvalue problem (16) Au- £«u = O. (17) Ldv(uit(T.O) + U(r, 0))u(0) = 0. Setting Ut(T,O)+ U(T,Z) = g(I'), = dg(I')v (ve onfit the dependence with respect to r in this section since r is fixed), we rewrite the probleln in the prinfitive variable. Hence we have to study the equation (18) AA°v + Ae' - (bv')' = A'v + (Cv)', (19) Lv(0) = (lw(0) ) :(0) =0 STAI3ILITY OF BOUNDAR LAYERS 2997 where v = , An(z) = dg(I'), A(z) = df(l'), b(z) = b(l'), and Ch z db(V)hI a. Note that we have the estimates for some M > O. where S(z) stands tBr Moreover, note that thauks to (H 1), (21) II(Cv)' = O. where Hw = w. Z Let us assmne that there exists a nonzero soluti.n of (18), (19); withom 1.ss of generality, we assume that (22) IIvll = 1. In this section, since we dcal with flmctions that take complex values, we denote bv u v the scalar product of C n, and by - the associated uorm. th«u define II,ll 2 = -0., (.) = ,«()-,,(.)dz. split the proof of the theorem into several lemmas. We will collect, all the estimates at the end of the section to reach our conclusiou. first give an energy estimate iu the saine spirit as in the totally parabolic case [2], [4] or in the pure Dirichlet's bouudary coudition case [12]: Lemma 4. Assm,e that v is a solution 4 (18). (19) that satisfies (22). Then. when is scieztlg small, we have the estimate (z3) n + l'll + l,(0)l 5 c. . (24) « + 11'112 + lw(0)l = cs(I,.(0)l = + I1'11=), (25) IZ, 1 c( + I1"11). Note th.at the first estiçmte (23) 9ives (26) e C3. Proof. first use the saine energy estimate as in the strict]v parabolic cse [2], [4] and the full Dirichlet case [12]. take the Hermitian product of (16) bv (in this section, we wil] denote S(V) by S for the sake of simplicity) and we take the real part, getting « (SA%, v) + e (SAc', v) - « (S(bv')', v) = e (SA'v. v) + Re (S(Cr)', v). 2998 F. ROUSSET Since SA is syminetric, we get Tae (SA-d, v) = > thanks to (tt6). Note that I(SA)'v,v)l <_ C + Next, integrating by parts, we bave 1SAv(O).v(O) ( (SA)'v, ,,) - - -((SA)'., ,) q- cl,(o)l 2 IV'llvl 2 Tac ( s(bv)', v) = -R « ( Sbd, ,') - n« ( ( S'bv', ,) - Tac Sb,' (0). v(O). Thanks to (H3'), we have nloreover, we bave « (Sb',') _> zlz'12; Tae Sbd(O) . v(O) = 0 thanks to the structure of the matrix b given by (H1) and (19). Using again (ttl) and (4), we have ( 1 0+° ) (('bd,.)) = (<b,z',z) < C ollz'll 2 + IV'llzl 2 for every q > 0 by using the Young inequality. Moreover, we have [(A'v, v)l _< C IV'I Ivl 2 dz and (27) I(S(o)',,)l = J0 +oe lJ0+°c ) < c IV'l MIz'l < C@ll'll 2 + IV'lll thanks to (H1) and (20). Çollecting these various inequalities, we have shown «ll.vl12+llz'l12+l,(o)l 2 Cllz'll2+C() IV'llzl2+C() IV'lM 2. To conclude, we first choose = , then we use z(0) = 0 through the inequality 14x)l + c[ + _ (28) IV'lll < xl"lll»'ll < Cllz'll and finalb; we absorb the terres Celle'Il l,d Ç(e) f IV'III i, th i prt 11'11 f -me,tb" n. hi proe (23). To get (24), we use (29) IV'll,l olV'lll'll + I(O)1 IV'l O(l(o)l + II,'ll). STABILITY OF BOUNDARY LA' ERS 2999 To prove (25), we also take the scalar product of (16) by St,, we take the imagi- uary part and we onlv use z,,, (s,,.", ) : -z ( Sb,', ') - Z,,, ( ( Sb)',,', ,,) C(llz'll + I1'111111)- get z, llll 5 C(ll,'ll + I1'111111 + allll) To conchlde, it suites to use (23), whi«h gives, in pa.rticular, I1'11 Call,,ll and the normalization assmnption (22). In the case of a pure Diri«hlet boundary condition, a weighted energy estimate on the hyperbolic part of the system (that is to say on the first n - r equations) w used in [12] to bound the terre This estinmtç was similar to the one used by (ioodman [3] for tire stabilitv of viscolls shock profiles. This was ecient because of thc upwind propagation. In out more general seting we use an energy estimato of "Kawashima's typo" [6], [9]. Lennna g. Assume lha¢ v is a solvtion of (18), (1.9) tat sat.sfies (22). Ten for scieal small . we have (30) II.'ll Proof. We use the nmtrix k givcn by (10). apply b to (16), we take the scalar product by v' and we take the real part. Using "e(kAv', v') = ((kA)*v ', v'), we get Here we have used the estimates (28) and (29). Using that kA ° is skew-Hernlitian, we have (kA°t ', v') E R since (a-A°v, v') : kA°v(O), v(O) - ((A°)'v, v) - (kA°, ', v) = -(A".'. v). Consequently, we have I«(a(A%, ¢))1 : Id )(.4 , ¢)1 S C«allll I1¢11. Siuce we have the estimate (10) ((A)''. ') 011'11 - ClI'II , we get IIv'll 2 < C(llz'll 2 + IIz"ll IIv'll + ne I11111'11 + all'll 2 + al'(o)12). and hence choosing r/> 0 sufficiently small, using the Young inequality and (9) we bave / (31) II'll 2 < c(,1)l, llz'll 2 + Consequently (30) is proved. (ne x)211,112 +- I1»"112 + al,(o)12). 3000 F ttOUSSET To end the proof of the theorem we would want to estimate []z"[[ with respect Lemma 6..4.ssne that is solution of (18), (19) that sati.sfies (22). Then. when 6 i,s stcientl sntall , u,e bave I1"11 (32) Pçoof. take the derivative of (16), getting the equation ",' + A,," - (b,'")' = O(IV'l)(l I + I"1 + I'1) + (C)". Thc proo[ is very similar to the proof of (23), in that we take the scalar product of t.he equati(m hy Sv and we do an integration by parts. The «boundary'" terres do hot vanish since d(0) does uot satis" the bOulMary condition (19). b just point mt that to lmund the terln ((S(Cv)",v') we also do 11 integratiou bv parts as iu (27) to get an estilnate independ¢,lt of 2.1. Proof of Theorem 3. ke now give the proof of Theorem 3. To conclude. we first have to eliminate z"(0) and v(0) in (32). We first express w'(O), lhanks to thc hyperboli« part of equation (18): ,.'+ .a, ,' + .a.=' = O(1"1)(1 I + I'l). where A = A. .4 . Note that we make a crucial use of (21). Since the boundarv is non«hara«teristic for the viscous svstem. A is nonsingular: moreover, thanks to (2g), (25), we have (33) We dedu«e I,'(o)12 _< c'(l,(o)l 2 + I»'(o)12 + 1,(o)1211v'll 2) (34) <_ C(l,,(0)l 2 + I-'(0)12 + llv'll 2) since thanks to (2g). we have w(0)] The next step is to estimate lz'(O). We use the classical Sobolev inequality (35) Iz'(o)l oe 211»"1111zll ,#llz"ll oe + llzll oe for every q ) 0. Hence it SllCeS to estimate z"(0) in (32). %7 use the parabolic part of the equation 4 ' O(v) + Aa,«' +. : -b," = O(IV'I)(II + Idl)+ 0(1111). We get. thanks to (23), (25), (34), and (35), 1 12 11,,112) (o) I"(o)12 c'(l(o)l 2 + ( + ,])11'1 + 11'112 + Next. we choose q such that C'q < 1, and we replace (34). (35), (36) in (39]. getting (7) I1"112 ç(lldll 2 + I1'112 + 1,,(o)12). çinally. «oll«ting (23), (30) and (37), w bave shown that (« )( - raz) + ( - c')ll«ll + ( - C)l.,,.(o)l 2 o. STABILITY OF BOUNDAIY LAYEIRS 3001 Hence if 5 is sufficiently small, this gives if T¢e A > 0, z = 0 and w(0) = 0. The hyperbolic part of the equati(m then t)e«omes a first-older ordinarv difl,rential equation involving only w: ,,' : (A)-(-X, ' + 0(1"1),') with the boundmT condition w(O) = 0. Conseuently we a.lso get w : 0. This ends the proof of Theorem 3. 3. NONLINEAR STABILITh In this section we prove Theorem 1. We use the form (3) of the svstcm. Setting u e = g(v e) and u app = g(vaPP), we have the two systcms J(Ve)t ÷ (f(ve)x = Lve(t. 0) = g. ,'e(o. *) = ,'o(.r) and Setting v E = v app + v (we Olllit thc dependence of t, iii ), We rcwritc Olll" lrollt'lll s A°i)tv + Ac%v - eO.(bO:v) = R G + 310 + ]111 + (38) where A ° b = M ° = AI 1 = 11 '2 = Note that v satisfies the (39) and the initial condition (40) L,(t.O) =0 (41) where v(O. x) = O. We «hoose C suffi«iently large such that Q <_ Ce N, 3002 F. ROIJSSET for solne large N which will be chosen later. To plove Theorem 1. we lise the classical çontinuous induction argmnent ([3], [7], [51, [9]). Let IlS define + «Oz(s)l[ = + «ll0,z(s) = d. + t,(s. 0)l = + «=10¢,(.. 0)1 = + «410,¢,,(t. 0)1= d.. Note that thanks to (39), (40), (4), we have (42) E(o) ç'e . Using the classical short-lime theory, we define r* = ,,p{ç« e [0. ç], oh,ti,,, of (3S), (39), (a0) e [0. r). E(t) e ' } Sll(.h that Vt where we choose Ni < N. There are wo possibilities: () T* = T, (2) T* < T. and E(T*) = e N. In the following, we show bv an energy estimate that we cannot be in the second case. This will show Theoreln 1. Let us define a() Z = Slip slip 00îI" t,-- . a ê te[0.T] 2>a>l,2kk0' Al first we need an elementary lemlna about the estilnates of the nonlinear quan- tities that arise in (38). Lemlna 7. Vi = 0.1. IOAI < c(10«vl)(1 -4-10,vl), STABILITY OF BOUNDARY LAYERS 3003 / Remark 8. '--t x '* Z also point out that, actually, we will hot usç the case o = 2,/3 = l in Lelnlna 7. ç now corne to tlw proof of oto nmin theorem l. In the proof. C stands for a number that is independent of ¢ but nmy dOl»end on T. Since bv classical Sobolov emtmddings, we have T* (T* 1Ozle and since by using the hyperbolic part o[ efluation (38) (thal is o say, the w component), the nonchmacteristic assmnption and Remark 8. we bave we gel the estimale (43) Ce,- Çonsequcntly, we choose N 6 and 5 < N < N. This allows us to use Lemma 7. Moreover, thanks to (43), we oi)tain })y contimfity ri'oto (H6) that (44) S,,,(v since Note that our smalhmss assmnption (9) and (6) imply that (45) p (z)+ () d: + () dz 5 C. z6N+ , , As for the spectral stability, there are four steps in the proof. Ve first lnake the energy estilnate of the totally parabolic case; next we lnake an estimate of Kawashima's type and an estimate on the space derivative of equation (38). The final step is to estimate the boundary values. For this, we replace (25) in the time evolutionary setting bv ai1 energy estilnate on the time derivative of the equation. At first, let us make the saine energy estilnate as for the totallv parabolic case [2], [4]. - (46) ,0 ÷°c a(ç)lvll0xvl _< CllOvll a(-)lvl _< c(«ll0.ll we easily get., after absorbing the terres CS]w(t. 0)12 by the ter,,, ct[w(t, 0)12 left-hand side, (48) Ot(SA°v,v)+a[u'(t,O)[ 2 where S stands for S(1 app ÷ ,). on the 3004 F. ROUSSET Note tlmt an estinm.te such as (47) is needed to bound the teInls (eSi:).(bOxv), z,), Next we replace the estinmte (25) of the spectral sta.bility bv an estinmte on the time derivative Ov. Since 0«¢, still satisfies the bomdary condition LO«v = 0. we can perform ¢he saine computation as previously on the rime derivative of (38). Thanks to Lemnm 7. we get (4) o,(sA%.o,)+lO,,,(t.o)l=+5dlO,.ll = the ideas of the computation have been use& We do hot give more details since ail \Ve just point out that we bave used (50) and that fo bound the terre we perform an integration by parts and use the block assunlption (H1) and (4). This ternl is then dominated bv C(ll,ll = ÷ 11,=1 + qllo,=ll =) for every q > O. , absorb the last factor by the terre flllO,.=ll . the left-hand side v choosing 1 suciently small. Note that for the moment, we do hot use an ineqlmlity similar to (46) to bound terlns Sllch as +oe a()[tt' 2 bomd this terre by expressing Ott', thanks to equation (38) and by using esti- mates such as (46) and (47). Then we get COIlleS frolll (Ç ' ) _< c I,.(¢.0)1 =÷-IIôx¢,ll = . STABILITY OF BOUNDARY LAh ERS 3005 Replacing (51) in (49) yields (5"2.) ,(S'A",,,.O,,,) ÷ lO,.,(t.o)l I c( « + I111 = + Ila,,ll = + (] + )lla=l + As for the spectra} stability, thç next step is to use the "Kawashima'" estimate [6], [9]. b aI,ply k(v (53) (kA°0,,..O,v) +(('- "' . - use the cruciM estimate (10). This Next we xvrite for every q > O, = I«(»O,.bO,.,..O,v) + < ('((« + )110.,'112 + llO,zlli.,,ll) C((e + d + )110,11 = + where here we bave used the blo«k structure assmnt)Iion (l[ l) and tlw 5omig in- equality. Using Lemma 7, the ung inequality, and (47) we bave (55) Bv the smne method, we ge simi}ar estimates for (kM °, O.u) and (kM 2. Ozt'). To handle (kA°&u. 0«v), we write Performing an integration bv parts in the lt factor al»ove, we get + (¢(kA°)t,.Otv) + and hence, since kA ° is skew-symmetric, 1 (O,(kA°v.Ov) - (Ot(k.4°)v.O«v) (kA°Otv. Ov) = + kA°(t. 0),9.0). O,,,(t. O) + (O(.A°),,, 0,.,.)). To bound (O«(kA°)v. O,v). we uoe - ( )MIO, vl d« thanks to (46) and (51). 3006 F ROUSSET This yields 1 o (56) (kA°Otv. O«v)=- for any q > 0 by use of the ung inequality. Consequently. collecting (56), choosing q sufficiently small and using our assumption (9) to absorb (e + )ll0vll in the left-hand side if is sufficiently small, we get from (53), + In conchlsion, it relnains to estilnate with respect to z and we perform an energy estimate similar to (2), but now the boundary terres in the integration by parts do hOt vanish. Using Lemma 7 and estimates such as (46), (47), and (50), we get (58) O,(SA"Ov,Ov) + llOll c + (1 + + Iou(¢. 0)1 + 10(, 0)1). Note that to estimate the terre (Ox12,âxv) we have perfomed an integration 1)y parts to avoid that terres involving IlOfll appear and that to estimate (SOAaOcv, Ov), we write f+ I(SOA°O,,Oæ)I and we use (51). As for the spectral stability, the next step is to estimate the boundarv values Ov(t,O) a,,d 0(.0). rt write th ,og, of (aS), 1 (59) 1o.(t. 0)l 2 for some q sufficiently smal] so tiret the terre wi]l be absorbed by g]e ]efg-hand side of (58). To est.image Ow(t, 0), we use the hyperbolic parg of the equation, the facg that the boundary is noncharacteristic and Lemma 7, getting the estimage IOw(t,0)l c(IR«(t,0)ff (60) c(l(t, 0)1 1 STABILIT OF BOUNDAR3 LA5 ERS 3007 To estimate O.zz(l, 0), we use the parabolic part of the equatioll. This vields (6a) 2lo=z(t.o)12 C(l(t.o)l 2+lo,.,(t,o)l =+«,llo.=zll + --0»1 + ( + , - Next, putting (59), (60), (61) in (58) gives the bound (62) Ot(SA°Ov,Oz ,) + «]]O=zll 2 ( ,,,,,, c 0 + (a + )1111 = + (a + )110.11 = + . Fina.lly, we «onsider (48) + «2(52) + «(57) + F«2(62) with F > 0 suffi«ientlv large and independent of . lntegrating ri'oto 0 to t, we get the estimate (63) IIz,(t)ll 2 + 2110z'(t)ll2 + 2ll0=(t)ll 2 + Ç «ll0z,(s)ll 2 + «31lOtz(s)ll 2 + l'«3lloz(s)ll 2 d.) ç IIz,()ll 2 + «2llO,,,()ll 2 d. + e N) thanks t.o (dl), (2), and (9). Note that we havc also used that N'.4" is positive definitc symmetri«. Thanks to 5ung's inequality, we bave for any q > O. Hen«e «hoosing q su«iently snmll su«h that Çq < 1 and then F C sucient}v large su«h that ç > , we finallv get «allO, xvll = + alO,.,(t, o)1 =. Next using Lemma 7. we perform estimates analogous to (49), (57), and (62} for Ottv, Ot«v and Ot«,z respeetively. Finally. we obtain the estimat.e E(t) <_ c(N + fotE(s)d.s). Therefore, by Gronwall's Lemma we bave :(t) < c« vt [o.w*]. 3008 F. ROUSSET Hence E(T*) < ¢N since N > Ni and hence T* = T. 1 EFER.ENCES [1] M. Gisclon, Étude des conditions aux limites pour un système strictement hyperbohque, via l'approximation parabolique, J. Math. Pures Appl. (9) 75 (1996), no. 5, 485-508. 97f:35129 [2] M. Gisclon and D. Serre, Étude des conditions aux limites pour un système strictement byberbolique via l'approxinatio parabolique, C. R. Acad. Sci. Paris Sér. 1 Math. 319 (1994), no. 4, 377-38-'2. NIR 95e:35119 [3] J. Goodman. Nonlinear asgmptotic stabilitg of viscous shock profiles for conservation laws, Arch. RationM hlcch. Anal. 95 (1986), uo. 4. 325-344. hlR 88b:35127 [4] E. Grenier and O. Guès, Boundary layers for viscous perturbations of noncbaracteristic quasilinear hgperbolic problerns, .1. Differential Equations 143 (1998), no. 1, 110 146. MR 98j:35026 [51 E. Grenier and F. Rousset. Stabil,tg of one-dimensional boundary layers by using Green's fttncttoï)s, ('.ondin. Pure Appl. Malh. 54 (2ri01), uo. 11. 1343-1385. MR 2003a:35126 [6] S. lçawashilua. Sgstcms of a hgperbolic parabolic type witb applications to the equations of magetohydrodgamtc«, Ph.D. lhesis. Kyoto University (193). [7] G. Kreiss and H.-(ï). Kreiss, Stablity of systems of viscous conservatzon laws, Comln. Pure Appl. Math. 51 (1.q98), no. 11-12, 1397 1424. hlR 2000c:35156 [8] T. T. Li and W. C. Yu. Boundary value problems for quastlincar hgperbolic systems. Duke University Malhematics l)epartlneut. Durham, N.C.. 1985. NIR 88g:35115 [9] C. Mascia a.nd N. Zumbrun. Stçbility of wscous shock profiles for dissipative symmetric hyperbolic-parabolic systers, Preprint (201). [10] A. lkIatsumura and K. Nishihara, La,ye-time behaviors of sobdions to an znflow problem in tbe balf pace for a one-dinensional system compressible viscous gas, Commun. Math. Physics 222 (2001), 149 474. MR 2002rn:76083 [11] F. Rousset, The boundarg conditions coming from the real vanishmg viscosity method, Dis- crete Continuous Dynamical Systelns (to appear). [12] lï). Serre and K. Zumbrun. Boundary lager stability in real vanishing viscosity limit, Commun. Math. Phys. 221 (20[)1), 267-292. [13] Y. Shizuta and S. Kawashima, Sgstems of equations of hyperbolic-parabolic type with appli- cations to the discretc Boltzm.ann equation, ttokkaido Math. J. 14 (1985), no. 2. 249-275. h'lR 86k:35107 [14] K. Zulnbrun. Multidimensional stability of planar viscous shock waves., Advances in the lheory of shock waves, Prog. Nonlilmar Differential Èquations Appl. 47. Birkhaser, Boston. MA. 307 516 (2001). (Ènglish}. hlR 2002k:35200 [15] K. Zmnbrun and P. Howard. Poitwise semigroup methods and stability of viscous shock waves, Indiana Unir. hlath. J. -17 (1998), no. 3, 741 871. NIR 99m:35157 ENS LYON. UhlPA (UMR 5669 CNRS). 46, ALLÉE DITALIE. 69364 L",ON CEDEX 0ï. FR.-kNÇE E-mail address: frousseteumpa.ens-lyon, fr Curret address: Laboratoire lï)ieudonné, Université de Nice-Sophia Antipolis. Parc Valrose, 06108 Nice Cdex 02, France E-mail address: froussetemath.unice, fr Editorial hlformation To be published in the Transactions, a paper must be correct, new, nontrivial. and significant. Further, it must t)e vell writtcn and of interest to a substantial numher of mathematicians. Piecemeal results, such as an inconclusive stop toward an unproved major theorem or a minor variation on a known rcsult, arê in general hot acceptable for publication. Papers submitted to the 7ra.sactions should exceed 10 published journal pages in length. Shorter papers mav be sul)mittêd to the Proceedings of the .4merican ll[atheroatical Societ.q. Published pages are the saine size as those generated in the style files provided for OE,v/$-I35FX or OE/,_ç-l\. As of Fel)ruary 28, 2003, the backlog for this j«mrnal was approximately 2 issues. This estimate is the result of dividing the number of manuscripts for this journal in the Providence office that have hot 3"et gone to the printer on the above date bv the average number of m'ti«les per issue over thc previous twelve months, reduced 1)y the numl)er of issues l)ublishcd in ff)m" months (the time ne«essary for editing and comt)osing a typical issue). In an eN»rt to make articles available as quiçkly as possible, articles are posted to thc AgIS wcbsite individuallv after t)roof is returncd from authors and belote apl)earing in an issue. A Consent to Publish and C«»pyright Agreement is rcquired bcfi»rc a paper will be published in this journal. After a Imper is a««epl'd for publication, the Providence office will send a Consent to Publish and C»l).vright .\greement to all authors of the paper. By submitting a pal)er to this journal, authors certify that the results have hot been submitted to nor are. they ramer consideration for publication |)v anolhcr journal, conference pro«eedings, or similar publication. Information for Authors Initial stlbnlission. Two copies of the pat)er should be sent directlv to the appropriate Editor raid the author sllould keep a coi»3". IF an edior is agreeable, an electronic manuscript prepared in TI)X or ECTEX nlav be submitte(l by pointing to an appropriate UFIL on a preprint or e-print serrer. The first page must consist of a descriptive title followed bv an abstract that summarizes the mticle in language suitable for workers in the general field (alge- bra, analysis, etc.). The descriptive title shoul(l be short, but informative: useless or vague phrases such as %orne renlarks about" or %oncerning" should be avoided. The abstract should be at least one conlplete sentence, and at nlost 300 words. Included with the footnotes to the paper should be the 201)0 Mathernoics Sbject Classiflcatio representing the primary and secondary subjects of the m'ticle. The classifications are accessible fronl . ams. org/msc/. The list of classifications is also available in print starting with the 1999 annual index of ll[aternaical views. The Mathematics SuSject Classification footnote may be followed by a list of ke!l words ad ptrases describing the subject matter of the article and taken from it. Journal abbreviations used in bibliographies are listed in the latest AIat- ern, a.ical Reviews ammal index. The series abbreviations are also accessible from .ams.org/publ±cat±ons/. To help in preparing and verifying references, the AMS offers MR Lookup, a Reference Tool for Linking, at . ams. org/mrlookup/. When the manuscript is submitted, authors should supply the editor with electronic addresses if available. These will be printed after the postal address at the end of each article. Electronically prepared nlanuscripts. The AgIS encourages electronica.lly lre ' mlsçrlnt ,rith et ....... çerence for ,A.A/[$-DTEX. To this end. the Society has prepared ,A.M$-L,TF author pa«kages for ea«h AMS publication. Au- thor packages include iustru«t.ious for preparing ele«trouic manuscripts, the .4MS Au.thoç Handbook, samples, aud a style file that geuerates the particular design specifica.t.ious of that publicat.iou series. Art.i«]es properly prepared using the X st.y]e file aud the label and ref commauds automatically enable extensive intra-document liukiug to the bibliography and other elements of the article for searchiug electrouically on the $b. Because ]iuking must ofteu be added manu- a]ly to electrouica]]v prepared nmuuscripts iu other forms of also redu«es the amouut of te«huica] interveutiou ouce the files are received by the AMS. This results iu fewer errors iu processiug and saves the author proofreading time. MS- papers also move more ecieutly through the production stream, helping t.o miuimize publishiug cost.s. MS-LAX is the highly preferred format of X. but author packages are also «vai]able in MS-. Those authors who make use of these style files from the begimfing of the writiug process will fluther reduce their own efforts. [amscripts pr«'pared e]e«trouicd]y iu I or p]aiu are norma]lv uot acceptable due to the high amouut of teclmical time required to insure that the flic will run properly throgh the AMS iu-house production system. LX users will find that Ex is the saine as EXTX with additiomd ('Ollllil&nds to simplify the typesetting of matheliiatics, and sers of l)laili should bave the foulidatiou for learniug Authors may ret.rieve ail author pa«kage ff'oto the AMS websie starting from . s. org/tex/ or via FTP to ftp. s.org (logiu onous. enter user- lmme as psword, aud type cal pub/author-info). The AMS Aathor Handbook and the bstruction Manual are available in PDF format following the author pack- ages liuk from .s.org/ex/. The author package can also be obtained free of charge bv seudiug email to pubs.org (Interuet) or from the Publicatiou Di- visiou, Americau Iat.hematical Society, 201 Charles Street, Provideuce, I 02904- 2294 USA. Wheu requesting an aut.hor packge, please speci- - or , Macintosh or IBI (3.5) format, and the publicatiou in which your paper will appear. Please be sure to include your complete mailiug address. At the time of submissiou, authors should iudicate if the paper has beeu prepared usiug MS-T&X or MS- and provide the Editor with a paper malmscript that matches the electrouic mamscript. The final versiou of the electronic uscript should be sellt to the Provideuce oce immediatelv afler the paper bas beeu accepged for publicatiou. The author should also seud the fiual version of the paper malmscript to the Editor, who will forward a copy to the Providence oce. Edigors will require authors to seud their electronicallv prepared mauuscripts to the Providence oce lu a timely fhiou. Electrouicallv prepared mauuscripts can be sent via email to pub-submits.org (Iuteruet) or ou diskette to the Elec- tronic Prepress Departmeut, American Mathematical Society, 201 Carles Street, Provideuce, HI 02904-2294 USA. When sending a manuscript electronicallv, plee be sure to include a message indicating in which publication the paper bas been accepted. No corrections will be accepted electronically. Authors must mark their changes on their proof copies and return them to the Providence oce. Complete instructions on how to send files are iucluded lu the author package. Electronic graphics. Comprehensive instructions oll preparing graphics are available startiug from www. oms. org/jourhtml/authors, html. A few of the nlajor requirements are given here. Subnlit files for graphics as EPS (Encapsulated PostScript) files. This iucludes grapllics originated via a gral)hics ap ..... r other COmlmter-generated images. If this is lier possible, TIFF files are acceptable as long as thev tan be opened in Adobe Photoshop or Illustrator. No lnatter what lnethod was used te produce the graphic, it is ncccssaly te provide a Imper copy te the AMS. Authors using graphics packages for the creation of elcctronic art should aise avoid the use of any lines thilmer than 0.5 points in wMth. lImy graphics packages allow the user te specify a "haMine" for a very thin line. Hairlines often look acceptable when proofed on a typical laser printer. However. when produced on a high-resolutitm laser imagesetter, hairlilms l»eCOlne nearlv invisible and will be lest entirelv in the final pl'inting process. Screens should be set te values between 15% and 85%. Screens which fall outside of this range are too light or too dark te print «orr,ctly. Variat tons of scrcens within a graphic should be n less tha.l 1(1%. AMS policy on making cllanges te articles after posting. Articles are posted te the AgIS website individually after proof is return('d frein authors and before appearing in an issue. Te preserve the integrity of ele«tl'mically lml»lisllcd articles, once an article is individually postcd te the AXlS wel»site i»ul net yct in ail issue, changes calmot be ma.de in place iii the Imper. llow«,ver, an "Atded aller posting'" section lllav be added te the Imper right l»cfore the l/efl'rences when lhere is a critical errer in the content of the lmp'r. The "Addcd aller posting'" sectitm gives the author an Ol»l»Ol'tUlfity te correct this type of critical errer lefl»re the article is put into ail issue for I}rilltillg and before il is then reposted with the issue. The "Added after posting" section relnains a permanent part of the paper. The AMS does net keep author-related information, such as aftïliation, current address, and email address, up te date after a pal»er is initially posted. Ollce the artMe is assigned te an issue, even if the issue has net vol been posted te the AMS website, corrections mav be ruade te the paper l\v submitting a traditional errata article te the Editor. The errata article will a.ppear in a future print issue and will link back and forth on the web te the original article online. Secure manuseript traeking on the Web alld via email. Authors Call track their manuscripts through the AgIS jollrnal production process using the personal AgIS ID and Article ID printed in the upper right-hand corner of the Consent te Publish ferre sent te each author who publishes in A,XIS journals. Açcess te the tracking system is available frein . ams. org/mstrack/ or via email sent te mstrack-query¢ams.org. Te access bv email, on the subject line of the message silnply enter the AgIS ID and Article ID. Te track lnore than one malmscript bv email, choose one of the Article IDs and enter the AgIS ID and the Article ID followed bv the word all on the subject line. An explanation of each pro<luction step is provided on the web through links frein the manuscript tracking screen. Questions tan be sent te tran-query¢ams, erg. TEX files available. Beginliing with the January 1992 issue of the Bttlletir and the 3anuary 1996 issues of Trarsactiors, Proceedings, Mathematics of Comp.uta- tiers, and the Jottrtal of the AMS, TEX files tan be downloaded from the AMS website, starting frein . a,ms. org/j ournals/. Authors without Web access may request their files at the address given below after the article bas been published. For Bulletin papers published in 1987 through 1991 and for Transactiors. Proceed- ings. Mathernatics of Computation, and the Journal of the AMS papers published in 1987 through 1995, TEX files are available upon request for authors without Web access by sending email to I±le-requestCams. org or by contacting the Electronic Prepress Department. American Mathematical Society, 201 Charles Street, Provi- doneo 11 fP)OOA_990 I lI A m, ....... ;hould iliclu<le the title of the paper, the naine(s) of the author(s), tho naine of the lmblicati«m in which tho paper h or will appear, and the volume and issue mlnbers if kn«»wn. The file will be sent to the author making the request aller the article goe to the printer. If the requestor «an receive Interner email, please include the email address to which the file should be sent. Otherwise please indicate a diskette [brmat and postal address to which a disk should be mailed. Note: Because production at the AMS soinetimes requires extra fonts and macros that are not ver publicly a»-ailable, files cannot be guaranteed to run through the author's version of without errors. The AMS regrets that it cann»t provide support to eliminate snch <,rrors in the author's T environment. Inquiries. Ara inquiries concerning a pal»er that has been accepted for pub- lication that calmot be a.nswered via the malmscript tracking system lnentioned above should l>e sent to tr-query+s, org or directlv to the Electronic Prepress Departmenl, Ameri«an lath,mati('al So«iety, 201 Charles Stl'eet. Providence. I 02904-2294 USA. Editors The traditional method of sul)mitting a paper is te send two hard copies te the appro- priate editor. Suhjects, and the editors associated with them, are listed below. In principle the Transactions welcomes electronic suhmissions, and seine of t he editors, those whose nallles appear 1)elow with an aslerisk (*). bave indicated that they prefer them. Editors reserve the right te request hard ce»pies after papers have been sul»nlitted electronically. Aut hors are advised te make preliminary inquiries te editors as te whether they are likely te 1)e al)le te handle submissions iu a particular electronic ferre. Algebra and algebraic geometry. KAREN E. SMIT|I. Department of Mathematics, Uni- versity of Michigan, Ann Arbor, iMI 4,119-1109 USA: e-mail: k«smith@umich.«du Algebraic geometry. DAN ABRAM()VI('I[, Deparlulent of Malhematics, Boston Univer- sity. 111 Cummington Street, Boston, MA 11221.5 USA: e-mail: abramovicbu.edu Algebraic topology and cohomology of groups. STE\VAHT PIIDDY, Deartment of Mathematics, Northwestern University, 2033 Sheridan Road, Evanston. IL 60208-2730 L'SA; e-mail: priddy@math, nuu. * Cornbinatorics. SERGEY FOMIN, Department of Mathematics, Èast Hall. ['niversity of Michigan, Ann Arbor. MI 48109-1109 USA; e-mail: fominumich. «du Complex analysis and geometry. D. H. PHONG. Departmen! of Mathematics. ('ohmlbia University. 2990 Broadway. New h ork, NY 1127-002.q l'SA: e-mail: phongma'ch, coumbia. «du * Differential geonmtry and global analysis. LISA ('..IEFFI|"Y. 1)epartm('nt of Malh- ematics, University of Toronto. 100 St. (;eorge Strcet. T,»ronto, ()ntario, Canada lklSS 3(13; e-mail: jaffray@math. 1;oron'co. adu Dynamical systems and ergodic theory. I¢OBEICT F. \VII,LIAM, Department of Math- ematics, University of Texas, Austin, TX 78712-102 USA: e-mail: bobOmath.utexas, edu * Geometric analysis, TOBIAS COLDING. Courant Institute. New hork University, 251 Mercer Street, New York, NY 10012 USA: e-mail: coldingOcis.nyu.edu Geornetric topology, knot theory, and hyperbolic geolnetry, ABIGAIL TtlOiMP- SON. Department of Mat.hematics, University of California. Davis. CA 95616-5224 ['SA: e-mail: /hompson@ma/h. ucdavis, edu Harmonic analysis, ALEANDER NAGEL. Department of Mathcnlatics, University of Wis- consin, 480 Lincoln Drive, Ikladison. \VI 53706-1313 USA; e-nail: nagel@ma*ch. isc.edu Harmonic analysis, representation theory, and Lie theory. IqOBEHT .I. STANTON, Department of Mathematics, Ohio State [rniversity. 231 Vest ltqth Avenue, Columbus. OH 43210- 1174 USA: e-maih stantonmath, ohio-s/a/e, edu * Logic. THEODORE SLANIAN, Department of Mathematics. University of Clifornia. Berke- ley. CA 94720-3840 USA: e-maih slamanmath.berk«l«y.«du Nurnber theory. HAROLD G. DIAMOND, Department of lklathematics, Universitv of llli- nois, 1409 WesI Green Street. Urbana, IL 61801-2917 USA; e-mail: diamondmath.uiuc * Ordinary differential equations, partial differential equations, and applied math- ematics, PETER W. BATES, Department of Mathematics, Midfigan State University, Èast Lansing, lklI 48824-1027 USA: e-maih bat«smath.msu.«du * Partial differential equations. PATF{I('IA E. BAUMAN, Department of Mathematics, Purdue University, West Lafayette. IN 47907-1395 t'SA; e-mail: baumanmath.purdu« * Probability and statistics. K RZYSZTOF BURDZY. Department of Mat hemat ics, Univer- sity of Washington, Box 354350. Seattle, %VA 98195-4350 USA; e-mail: burdzy@ath. ashington. * Real analysis and partial differential equations, DANIEL TATARU, Department of Mathematics, University of California. Berkeley, CA 94720 USA: e-mail: tatarumath All other communications to the editors should be addressed o the lklanaging Editor, WILLIAM BECKNER, Department of Mathernatics, University of Texas, Austin, TX 78712-1082 USA: e-mail: beckner@,ath.utexas, edu MEMOIRS OF THE AMERICAN MATHEMATICAL SOCIETY Memoirs is devoted te research in pure and applied mathenmtics of the saine nature as Transactions. An issue consists of one or more separately bound research tracts for which the authors provide reproduction copy. Papers intended for Mernoirs should normally be at least 80 pages in length. Memoirs }las the saine editorial committee as Transactions: so such DaDers shofld t)e add,-o,:,,¢l t ..... r the editors listed al)ove. TNSACTIONS AMERICAN MATHEMATICAL SOCIETY TANSACTIONS " I Journal overview: This jour- nal is devoted to research arti- cles in ail areas of pure and applied mathematics.To be pub- lished in the Transactions, a paper must be correct, new, and significant. Further, it must -«,«,,,,, be well written and of interest . ,,', ............ to a substal number of information: I .'"J°r?'°":X I ' TRANSACTIONS ., inloratlon an ] o»t . befundnme - " Submission page. I , AZDED The AMS has released enhanced versions of its electronic journals. These upgrades improve usefulness and relevante for both journal subscribers and journal authors. A 30-day free trial is available to corporations and institutions. A downloadable Free Trial Form is available at: www.ams.org/ customers/ejournaltrial.pdf. Contact AMS Membership and Customer Services, 201 Charles Street, Providence, RI 02904-2294, USA; phone 1-800-321-4267 or 1-401-455-4000 worldwide; fax 1-401-455-4046; email: cust-serv@ams.org. Note:A signed license agreement is required for AMS electronic iournal subscriptions.A newly updated and expanded agreement tan be found online at http:llwww.ams.org/customers/jour-license.html. TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY CONTENTS Vol. 355, No. 7 Whole No. 818 July 2003 Borislav Karaivanov, Pencho Petrushev, and Robert C. Sharpley, Algorithlns fi)r nonlinear piecewise polynolnial at)t)roximation: Theoret- i('al aspects ......................................................... 2585 J6rg Brendle, The ahnost-(lisjointness mnni)er may have countable ('otimdity ........................................................ 2633 Alina Carmen Cojocaru, Cyclicity of Chi ellil)tic «urves niodulo p 2651 Tonghai Yang, Ta.vlor expansion of ail Eiscnstein serics .................. 2663 Eric Freenlan, Systenis of diagonal Diol)hantine ilwqualities ............. 2675 Francisco Javier Gallego and Bangere P. Purnaprajna, On the ean()ni('al rings ()f ('overs ()t" sm'faces of minimal degree ............... 2715 H. H. Brungs and N. I. Dubrovin, A classification and eXalliples of rank on(, chain «lomaius ................................................... 2733 Donald W. Barnes, ()n the Sl)e('tral sequelwe ('onstru«tors of Guichardet and Stefan ....................................................... 2755 Steven Lillywhite, Folmalitv in an equivariant sctting ................... 2771 Neil Hindlnan, Dona Strauss, and Yevhen Zelenyuk, Large re('tangular semigroul)S in St()lie-Cech ('onq)a('tifica.tions .......................... 2795 Takehiko Yamanouchi, Galois groul)S of quantuui group actions and regularity of fix('d-l)oint algel)ras ..................................... 2813 Boo Rira Choe, Hyungwoon Koo, and Wayne Smith, Çoniposition Ol)('lators a('ting on holoniorphi(" Sol)olev spaces ..................... 2829 B. Jakubczyk and M. Zhitomirskii, Distributions of corank 1 and their characteristic vector fiel(|s ........................................... 2857 E. Boeckx, Wlien are the tangent Sl)here i)undles of a lielnalmian manifold re(hicil)le'? ........................................................... 2885 Henri Colnman, Criteria for large deviations ............................ 2905 Seung Jun Chang, Jae Gil Choi, and David Skoug, Integration by parts fonnulas involving gelleralizcd Fourier-Feymnan trallsfornls on flill('tioll sl)ace ................................................................ 2925 Michiko Yuri, Thermodynamic formalisni for countable to one iklarkov systelnS ....................................